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ABSTRACT 



In this final volume of a three- volume series on the 
construction of self-instructional systems, the focus is on various 
aspects of post-construction control and improvement. As the first 
step in this phase, the process of polishing the material by checking 
for readabil ity, integration, retention, and moti va tion is described 
and some of the aids to be used in this checking phase are suggested. 
A description of empirical procedures is included to emphasize the 
need for systematic revision techniques. Some examples are presented 
of relevant variables and systematic mapping devices which aid in 
evaluation^ Combinations of revision criteria are reviewed and a few 
special measures to reduce student boredom are suggested. The final 
task in program construction, compiling a program manual, is detailed 
with a catalog of the categories of information common to any program 
manual. A check list is provided to facilitate the assessment and 
comparison of existing programs. A glossary defines terms used in 
programed Instruction and educational technology. See also volume one 
(EM 009 072) and volume two (EM 009 073). (JY) 
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Ake Bjerstedt 



The construction of a self-instructional system can be seen as a work 
process with three main phases: (l) System analysis: Preparatory 
work; (2) System synthesis: Construction of a preliminary system 
version; and (3) System modification and evaluation; Post-construc- 
tion control and improvement. - While phases 1 and 2 in the program 
construction process were dealt with in earlier reports (Didakometry, 
Nos. 30 and 32), the present survey focuses upon phase 3, discussing 
various aspects of post-construction control and improvement (such 
as the polishing and checking phase, the general procedure of 
successive evaluations, various evaluation criteria and evaluation 
aids, compiling the program manual). - Terminological and biblio- 
graphical appendices are included. 
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12. POLISHING AND CHECKING PHASE 



When the various instructional units have been constructed, a renewed 
examination is made of the unit sequence. It is, of course, important to 
check that notliing important has been forgotten, and tha^ the various 
parts of the program have been put together smoothly. But there are 
many other factors that the programmer should consider before starting 
empirical experiments with his material. In this chapter, some of these 
points of view will be briefly discussed, and various aids (designed to 
help the programmer in this checking phase) will be described. 

We will first group our comments on checking around four problem 
areas: (1) facilitation of communication (readability), (2) integration, 

(3) retention, and (4) motivation. We will thereafter devote a special 
section to various aids for systematic mapping and checking procedures. 

12.1 FACILITATION OF COMMUNICATION (READABILITY ) 

Those responsible for planning and writing the study material do not, 
perhaps, always notice the difficulties o ambiguities in their formula- 
tions. On the other hand, someone who jtas not been directly involved 
is aware almost immediately of such things in his capacity as recipient 
rather than dispenser of information. The final proof of the efficacy 
of the study program as a communication medium is, naturally, 
revealed in its effect on representatives of the student group. Ordinari- 
ly, however, considerable effort can be saved by carefu examination 
in advance. 

Someone familiar with programming, but not connected with the par- 
ticular project, reads it through, therefore, with the purpose of simpli- 
fying it and pinpointing obscurities. It is usually desirable that two 
others take part in this preliminary study. An expert on the subject 
makes a detailed examination of the factual material and also evaluates 
how well the course material is covering the ground required. The ma- 
terial is then read for a language control by someone who has both a 
good writing style and experience of the characteristic reading ability 
of the students in the target group. If the program contains special 
types of aids (pictures, sound effects, etc- ), it is often necessary to 
call upon other experts to assure the best possible technical use of these 
aids. In this section we shall briefly present some ideas on one of these 
aspects of revision, namely, the question of readability or - in more 
general terms - communicability. 
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Ainong the n^ain questions the reviser should ask himself are these. 

(1) Is the presentation of material as far as possible adapted to the spe- 
cial qualifications of the students ( student-adapted communication ) ? - 

(2) Is the thought content expressed with an optimal degree of precisiqn ?- 

(3) Does the physical structure facilitate as much as possible a meaning- 
ful focusing of attention ? 

One study (Grace, 1963) indicated that programmed material differed 
significantly from non -programmed material in respect to several fac- 
tors which are usually considered to be connected with readability. The 
programmed study material used, on the whole, shorter words as well 
as a higher percentage of the more frequently used words in the language, 
and new words were introduced at a slower rate. Even if the results of 
this particular study may not be entirely reliable (since wc do not know 
much about the comparability of the two groups of texts used in other 
respects), the indications arc that program writing as such tends to in- 
fluence the author to a use of words better adapted to the language abi - 
lities of the student. This is quite natural, since the author, unlike the 
writer of textbooks (and through the presentation method itself with its 
constant demands for an answer), is not allowed to forget that he is 
speaking to a specialized audience with a claim to attention. 

The demand for student-adapted communication does not mean, na- 
turally, that a whole program can be c onfined within the boundaries of 
a minimum language area which all students at the beginning of the course 
can be expected to command. But it means that one must start f rom that 
level and must make sure that new term.s arc not introduced without an 
explanation understandable to the students (that is to say, an explanation 
v/hich uses for its definition language which can be understood by the 
students) and at a pace accomodated to their' abilities (i. e. , not so rapid- 
ly that the students cannot, when tested, demonstrate their ability to 
absorb the now words). In this connection good use can be made of the 
original student analyses. For this purpose it maybe desirable that the 
student analyses contain recorded samples of the normal conversation of 
the test group as well as sample results showing their vacabularv com- 
prehension (passive reading comprehension). Although it maybe difficult 
in the first version always to keep these factors in mind along v'-th ideas 
for presentation of the content, the final reviser can usually easily 
observe and correct the more outstanding violations of the demand for 
adjustment to the language needs of the students (when his attention is 
directly fixed on this particular problem). 
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In this connection it might be well to repeat our previous warning 
against routine application of different readability formulae which deal 
with average values (for word length, word frequency, etc. ). In a more 
exact and psychological sense, readability can only be studied in rela- 
tion to an individual. It is an individual matter which, moreover, is in 
a state of flux. What is ’’readable" for one student in the last phase of 
the program would in many cases have been "unreadable" for that same 
person at the beginning of thci program. A routine, average appraisal 
gives a relatively rreaningless result in these; eases. 

The question of the precision of the expression used must always be 
viewed in relation both to the students' receptivity to a certain type of 
information and to the goal which lias been set. It is not always a case 
of striving for maximum precision, but rather an optimum degree of 
precision in relation to the circumstances and to the objectives decided 
upon. The risk fox excessive accuracy (for example, scientific exact- 
ness in a ease where it docs not fill a need) is considerably less - 
judging by published programs - than the danger of inadequate precision 
caused by haste or poor linguistic marksmanship. It is c ften a question 
of the hazy use of equivocal abstract words, where the sim.plcst ling- 
uistic revision is a transition to a greater concretion (and possible 
exemplification). 

At the present tim^c we do not know enough about the best niethods of 
achieving a well-balanced focusing of attention. NaturaMy, essential 
facts should not be allowed to disappear in a cloud of verbal mist. Long 
didulcs arc often improved by cutting, but a routine pattern cannot be 
used, Didulcs should be regarded as information units. It is not a question 
of cutting an excess number of words, but of weeding out those which 
are not important to the context: irrelevant and nneaningless filler 
phrases or details which do not contribute to the final objectives. It is 
possible to shorten long didulcs by (a) cutting down the amount of detail 
in the examples given, (b) replacing some verbal explanations with 
pictures or diagrams, and (c) incorporating some didulcs in the colla- 
teral material in the appendix. In many cases it becomes evident after 
a closer look at the material that it is advantageous simply to divide a 
longer didulc into several shorter ones. 

Most likely, however, an extrem o condensation is not the ideal solu- 
tion* The interaction between student and program can be regarded as a 
play of alternating tension and relaxation. The relaxation periods need 
Q not be long, but they should not disappear entirely. If every single word 
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is loaded with meaning, the tension can be unnecessarily tiring, and 
the focal moment is not delineated as a clear figure against a back- 
ground. Individual differences in reading habits for various types of 
texts should have some bearing on this, and student analyses could 
offer some insight into the problem. University students who change 
from a scientific course to liberal arts or vice versa usually experience 
significant difficulties in readjustment which, among other factors, 
should have some relation to different "density expectation", 

In this play between tension and relaxation, the points of tension 
should be represented by the cardinal points in the content and not be 
language difficulties (expect possibly in regard to terms for just these 
cardinal points in the content). Some idea of the density of content can 
be derived from an analysis of the percentage of occurrence of 
"structural words" ("non - significant" words such as articles, con- 
junctions, prepositions, etc. ) and "repeated v/ords" ("significant" 
words which have already occurred one or more times within a given 
section). The lower these percentages, thv- fewer the "rest periods", 
and the greater the probability that the student will become tired be- 
cause of the lack of balance in the relation of tension to relaxation (too 
great tension caused by too little word redundance). The cardinal 
points and new terms should be allowed to stand out against a more 
neutral background of familiar material which does not invite tension. 
Obviously this is a question of general principles which can be difficult 
to convert to specific rules for concrete examples without extensive 
trials. We know hardly anything more exact about the optimal values 
for different ages and different material. In general it can be said that 
an interesting field is opening up for teaching -related attention research. 

A language check should also cover accuracy and consistency (is the 
sentence structure effective, punctuation intelligible, use of abb reviation 
uniform, etc, )? This aspect, liowcver, is too self-evident to require 
any further discussion. 

Program writing demands a great deal of the author; He should be 
able to think clearly on an abstract and c omplex level (in order to carry 
out effectively what has been defined by means of the terminal de- 
scriptions and clarified thi-ough the material analyses), and, at the same 
time he must communicate clearly on a cone rete and simple level (to 
reach his students effectively). We seldom find anyone who has maxi- 
mum effectivity on both pianos, and for this reason teamwork and 
chocking should play an important role as correction factors. 

t 



iZ.Z INTEGRATION 



Writez’s of programs have often been influenced by theo*^ies with a be- 
havioristic tinge, where analysis of sub -components in observable be- 
havior has played a leading role. Psychologists and educators who hold 
different theories criticize the program writers because they ”atomi2;e" 
learning and because they take altogether too little interest in the 
‘’cognitive structure” or ’’Gestalt” of knowledge, that is, in the inte- 
gration of the parts into larger meaningful wholes. 

These differences of opinion have been clearly expressed at some 
international conferences devoted to problems of programmed instruc- 
tion. On the one side we often find the American progra.nmers, many of 
whom base their reasoning on a stimulus -response theory of pragmatic- 
behavioris tic type, sometimes considered to be fairly self-evident, al- 
though to some people somewhat hazy in detail. To them the behavioral 
terminology is usually the natural rr.cans of communication, while 
expressions like ’’cognitive experience”, "comprehension”, or 
"cognitive structuring” arc regarded as suspect remnants out of the 
philosophical past of psychology. On the other side wc often sec some 
psychologists from Central Europe, eager to learn the principles and 
technical design of programming, but often with a negative attitude to 
the "atomized” terminology, which to them sounds like the old psychology 
of association, once and for all - as they tend to think - jlown sky high 
by the Gestalt psychologists. 

Such disparities arc often interesting, and it should be a definite 
advantage for the development of programming technique if its 
principles arc gradually studied and worked over by persons with more 
varying theoretical points of departure. Nevertheless, one has the feeling 
at times that the differences arc more verbal than actual. If the debaters 
can only be persuaded to climb down from the level of general and verbose 
theorizing to the level of operational descriptions, they can often agree 
on facts. The differences reflected are, in other words, in many cases 
more to bo found in ’’symbol environment” and habits of thought than in more 
signficant divergences of view about the essential bases of facts. 

Thics docs not mean, however, that programming in certain forms 
cannot run the risk of giving an entirely too "atomized” knowledge with 
too few possibilities for the students to discover greater coherence and 
to achieve a perspective on that which has been learned. The programmer 
should under all circumstances be on his guard against these pitfalls 
(v/hich certainly can be found even in other forms of instruction in our 
schedule -bound school world). 
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In order to facilitate 'Hnteg ration” and ”structuring” of knowledge 
and to avoid that what has been learnt is experienced as unrelated pieces 
of mosaic, the programmer can use some of the following working me- 
thods: 

( 1 ) Preliminary organization and other supplementary information 
to ex p lain procedure (”meta -didulcs”) . When introducing a new topic, 
it is often advisable to build a minor unit sequence into the program, 
the function of which is to give the student a preliminary idea of the 
goal and of the s c op e of the study material. A method of this kind can 
also be used for smaller task sequences* Ordinarily such organization 
is of value both for the effectiveness of learning and for the general 
learning attitude. The student should know where he is going. The pro- 
grammer can also ”speak directly” to the student about the nature of 
the work under process in other ways, for example, by relating specific 
learning points to the ultimate goal of the course, pointing out special 
reasons for combining certain learning units, etc. Principally for what 
seems to be ’’puristic -esthetic ” reasons, such extraneous comments 

to the students have sometimes been avoided. Of course, they can break 
the continuity of the flow of information material (in about the same way 
as the actor^s aside to the audience can interrupt the rapid repartee on 
the stage). There is, however, good reason to assume that the intro- 
duction of such second -Older communications (’’communication about 
the communication” or ”mcta -didulc s ”) can have some bearing on the 
ercperienccd integration of the learning area as •well as on the work 
motivation of the student. 

(2) Successive linking of detail (avoidance of discor cinuous jumps). 
One important point in our checking phase is naturally a renewed study 
of instruction sequence, the individual mic ro -s equenc es (designed to 
teach special concepts or concept relations) as well as the total macro- 
sequence (with review of both the grouping of the subject areas and 
possibly the placing and character of branches). It is naturally important 
to make sure here that no significant item necessary to achievement of 
the goal is forgotten. But it is also of importance (especially with a view 
to furthering integration) to determine whether the various parts of the 
program are joined together smoothly enough and without disconnected 
jumps. In most types of courses it is both desirable and possib3c to tie- 
details together to form a meaningful '’associative net”. This occurs 

Q chiefly at the transition from didule to didule and at the transition from 
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one ri^icro-s equcnc c to another. By making sure that the new tasks con- 
tain certain elements from the previous ones ("connecting elements") 
we get smoother transitions, greater probability of correct answers 
and better -integrated knowledge. It is, however, difficult to give any 
general rule of thumb for how this is best accomplished. 

(3) Special assignments for furthering integration. Besides emphasis 
on the successive linking together of single details, it can be fitting to 
introduce from time to time special, more demanding recapitulation 
problems. Concepts and operations, which have been introduced in diffe- 
rent connections, can then be brought together in more comprehensive 

a ssignments. 

The possibility of working with program supplements, to which re- 
ference is made in the separate didules and to which the student is 
referred for information and compilations should never be forgotten. 

Such "supplements" can, for example, consist of maps, diagrams, 
tables or simply a summing up of verbal information. The assignments 
which are given in this connection can be directly aimed at integrati( n. 

It can often be helpful with shorter test sections (assignment se- 
quences without new information and without aid - stimulation) at regular 
intervals, perhaps after every 20th of 40th didule. If these are well 
constructed, they can undoubtedly contribute to knowledge 
integration. They can, moreover, perform^ other important functions. 
They probably increase retention and they give the student a feeling 
that ho really has learned something (which he does not always believe 
when progress is easy!) 

(4) Didule patterns with integration -aim. Not only can the single 
didule be so formed that it promotes integration of comiponent knowledge 
derived from earlier didules, but it is often possible and essential to 
arrange the single didules in an internal sequence so that the integration- 
aim is aided. Some potential procedures for accomplishing this which 
would be worth a more systematic testing for effectivity can be mentioned 
briefly: 

S’* Counterpoint sequence . Two content units which are to be linked 
together in the student's behavior repertoire are alternated systematically 
during the practice phase. 

Example: etc. 

(Here and in the following the slanting line is used to indicate the didule 
boundary and the letter C the basic subject-matter unit. ) 

er|c 
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b. Superimposed sequence ^ Two subject-matter units which are to 
be combined with each other in the student'^s repertoire can be or- 
ganized so that the latter unit is introduced just before that point where 
discussion of the earlier unit is completed. 



Example: C /C /C /C /C /C /C /C /C /C /C /C etc, 
^ x' x' x' x' Y ^ y x' y y' y y 



c. Accumulated summary . When the material units are part of a 

larger complex (build a cohesive total structure), but still must be 

introduced in stages because of the level of difficulty, every new didule 

can be arranged so that the part -element 5 which have been introduced 

earlier can be repeated at the same tirre as a new element is introduced. 

That is; If the student'^s terminal behavior repertoire is to contain the 

complex C C C C , the didule sequence can look like this: 

^ X y z u ^ 

C/C &tC/C h C h C /C h C 
x' X y^ X y z' X y z u 



d. Chronological accumulation of temporal behavior sequences can be 
seen as a special application of the general principle of accumulative 
summary, that is, the particular case in which a behavior chain based 
on time order is succf ssively built up in chronological order, as 
follows: 

C^/C^ C 2 ^/ C ^ ^ ^ ^ ^3 ^ ^4 etc# 

Many behavior scries arc of typical temporal character, that is, they 
form a definite chain of actions which occur in a characteristic order. 
Examples are working with more complicated tools and machines as 
well as fault localization on different types of apparatus. Instead of 
first learning every part-subsection separately and then trying to put 
together the small bits to form a whole ("delayed integration"), it 
should often be more natural to practise sequence from the beginning 
in the manner described above. 



e. Reversed accumulation of temporal behavior sequences. The 
temporal sequence should of course always be practised in the natural 
direction of association, Wc train, in other words, C^; not 

C^, On the other hand, it is quite feasible to accumulate the sequences 
"backwards" instead for "forwards". Instead of the didule series: 



C 

C 

C 

C 



1 

1 

1 

1 



& C2 

2 3 4 ' 
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in which is thought to represent the natural end link in the behavior 
chain, the following didule series would be used: 

C3&C4 

C^, & C3 & C4 

C. & C, & C, fk c. 

12 3 4 

The advantage of the latter arrangement can be that the student often 
derives greater motivati- • from being allowed to perform the terminal 
phase of a behavior seri. .t an earlier stage as well as to see each 
practice series finished off with the goal clearly attained. Especially 
in those cases where the terminal phase of the behavior series is one 
in which the result stands out clearly (gives a definite, even dramatic 
demonstration of goal achievement) this type of sequence arrangement 
can be of value. (This method or procedure has been particularly 
advocated within the frame of the so-called "mathetical approach" re- 
presented by Gilbert and his disciples; see, for example, Gilbert 1969; 
Pennington & Slack, 1961). 

) Adaptation of study material to divergent study behavior mo- 
tr^te d by diverging initial repertoire or aims (with gains in integration 
when used for review). Our ordinary textbooks and handbooks can be 
used in many ways. Besides being successively learned as course li- 
terature, they can be "skimmed" or "read diagonally" by someone who 
has already mastered the principal contents of the subject (in which case 
"skimming" provides a quick survey and the possibility of discovering 
any new points). They can also be used as reference literature by those 
who have a more specific end in view than that of learning the subject 
as a whole: by use of the table of contents or the index, it is easy to 
look up a special part which is relevant to the current object without 
studying the other material in the book. One criticism of programmed 
study material has been that the specialized form (for example, the 
fixed arrangement of "small steps", arranged in "scrambled" order or 
in horizontal "bands") makes it difficult or impossible to use the study 
material in a flexible way, and that the reader is bound to using it for 
only one single raading method - according to the rigid pattern of care- 
ful memorization. This me ans too that those who want to brush up their 
knowledge after a period of time and once more get a comprehensive 
view of the material which has been covered have difficulty in doing so - 
unless they take the time-consuming way of again going through all the 
material in the prescribed order, 

12 
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Anyone who tries to read through a finished program for a familiar 
subject area can easily be persuaded that these objections arc valid. It 
is true enough that the programmer can defend himself with the argument 
that the program is designed for persons with a certain type of initial 
repertoire and is meant for a certain type of goal, and that because of 
this the programmer has no reason to take into consideration the fact 
that persons with a different initial repertoire (greater knowledge) and/ 
or different goal (covering only certain parts of the field) think that the 
study material is not practical. 

On the other hand, it can reasonably be argued that difficult survey- 
ability, with the rigidity in usage which follows, is often greater than 
necessary and that the writer could by relatively simple means make 
the total structure considerably more transparent - to the benefit of, 
among other things, later reviews for improving retention. Even though, 
naturally, the primary usage should determine the presentation and 
arrangement, the secondary usage (as review or reference book, for 
example) should not be made more difficult than necessary. 

One technique which could easily be used to increase the structural 
surveyability is that of working out detailed tables of contents ^vith 
references to didules as well as drawing up a list of essential terms 
and ideas (also with references to didules), and also printing the inte- 
grating main sections (preliminary introductions and outlines, for 
example) in italics, in another color, or emphasizing them by some 
other means. Ordinarily it is an advantage if the main sections ("chap- 
ters") arc clearly separated from one another by page- It is usually 
easiest to set up separate chapter sections in programmed books with 
vertical progression, but this can very well be done in programmed 
books of other types. 

12. 3 RETENTION 

The resistance to obliteration of that which has been learned is naturally 
affected by many factors. Several of the points on readability and inte- 
gration which were presented above are also of importance in connection 
with retention. For example, the well-integrated material has - other 
things being equal - greater resistance to obliteration than the laot so 
well -integ rated. In this section wo shall confine ourselves to summari- 
zing some brief rules for repetition: 
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(1) Avoid too little repetition! Many a programmer includes too 
little repetition since the construction of repetition assignments is 
often boring for him and - since he himself is so familiar with the ma- 
terial - may also seem unnecessary. The student, on the other hand, 
who meets for the first time a number of new ideas and associations, 
is normally in great need of repetition. Retention of that which has 
been learned depends, naturally, to no small degree, on the "over- 
learning” which takes place in the instruction set-up. In this connection, 
however, the time-lag between the instruction occasion and the termi- 
nal situation must be taken into consideration as well as the student'^s 
opportunities for natural repetition or practice during the intervening 
period. Any possible dangers of interfering factors ("competing behavior 
patterns”) must also be noted. As a guide for these judgments, the initial 
studentanalysis is of value and also the "expanded terminal chart” worked 
out in conjunction with it(cf. discussions on initial system analysis). 

(2) Vary repetition ! Routine drill with direct repetition of the same 
assignments is seldom desirable, partly because going through the 
matoxial in this manner can easily become boring, partly also because 
the programmed sequences are worked out for those who have no ele- 
mentary knowledge and when repeated must be assumed to function less 
effectively. The practice should instead consist of illuminating and 
applying the acquired conceptions in new contexts or with different 
examples. It is often a good idea in this connection to integrate repe- 
tition of old material with the introduction of new. In this way repetition 
monotony is avoided, the area for meaningful associations is increased, 
and desirable generalizations are facilitated. 

(3) Spread repetition ! As has been noted above, it is true that a con- 
centration of learning can be effective from the point of view of speed 
but at least in the case of more comprehensive instruction material and 
long retention periods a certain spreading out of repetition should be 
desirable. It is thereby possible, for example, to treat the same idea 
first in a series of consecutive didules and later to allow the idea to 
turn up again at certain intervals (perhaps shorter in the beginning and 
successively with greater infrequency later on). A "distribution plan" 
could then take the principal form sketched below in Figure 1 2. 2 below. 

(4) Individualize repetition as much as possible! The most usual 
procedure, that of allowing the student to pass from one assignment 
to the next whether he has answered correctly or incorrectly, without 
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returning to it later, has certain disadvantages. As mentioned above 
in our discussion of various '’flow models”, a "rc-dirccting” to 
differentiated repetition (of assignments answc'^'cd incorrectly at an 
earlier stage) may sometimes be worth recommending. Skinner^s disc- 
machine (also referred to above) functions in just that way, and Holland 
& Porter (1961) were able to demonstrate that such a method can improve 
the final result of the student. These methods arc especially suitable 
for learning ’'subject-matter items” (as, for example, in learning voca- 
bulary in a foreign language), where this differentiated rc-direction 
does not break the continuity. Those types of machines which function 
according to the ” skimming’' -principle can be useful here, but simple 
card collections are often equally effective. In these cases every didule 
can be noted on an assignment card (with the answer on the reverse side). 
Those questions which the student answers incorrectly can be placed by 
him in a special pile. He can then easily go through these once more 
(or as many times as is necessary until he has performed all assignments 
cor rcctly). 




12, 4 MOTIVATION 

Reading a long, continuous text without ''built-in” work assignments or 
listening to an extensive oral exposition can be rather monotonous for 
many students. The result can be increased fluctuation in attentiveness 
with an accompanying reduction in the effectiveness of the instruction. 
Some of the increase in cffcctivity which in miany eases has occurred 
in conjunction with the use of programmed study material seems to have 
some connection with the less monotonous work dynamics in the didule 
system (rather than with the sometimes too greatly stressed reinforce- 
ment effects of feedback presentation). The student experiences a constant 
task variation: 

(a) he rads the informiation and the demand for an answer 

(b) he works through to an answer 

(c) he writes down the answer 

(d) he checks his answer and prepares to go on to the next assignment. 

The dynamic variation, the play between tension and relaxation (which 

we discussed in connection with readability), can here perform an im- 
portant function, and the differences in the activities listed under a,b,£, 
and d are probably of importance. It is a reasonable hypothesis that 
points a and b ordinarily signify a rising curve of tension (with the summit 
at b), while points £ and d represent a certain psychic relaxation during 
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the phase G with more predominant motor activities (writing and machine 
manipulation). Both the subjective experience of the dynamic alternation 
and posGiblc psycho -physiological consequences whould be worth a 
Special research effort. 

But even if the programmed study material becauGC of this inherent 
alternation of tasks is less monotonous than a running text or a con- 
tinuous oral presentation, it is quite natural that the student can ex- 
perience a certain monotony also in the programmed study material 
if, for a longer period of time or in several different connections, he 
works with the same type of Gmail presentation units (as long as we 
have only a few and/or short programs there is relatively little danger 
of this). Several of the measures mentioned earlier for promoting the 
building of an integrated field of knowledge for the students can also be 
useful in reducing the dangers of monotony: for example, work with 
integrating supplemental material in appendix form or going thorugh 
built-in test sections. Other methods of counter-acting monotony can 
be to alternate types of answer (certain sections perhaps without demand 
for an explicit answer), or to incorporate more comprehensive student 
tasks (directing the students to various source material; a technique 
termed above "the intermittent model*'). There is a risk that, through 
exaggerated respect for one special method, we fail to try the united 
effect of different methods. We must keep in mind to avoid this risk. 

That the experience of achievement is of great importance for the 
student^ s motivation should be indisputable, but in reality the problem 
is more c omplicated ‘than some programmers usually de sc rib e it. We 
must take into consideration the great individual variations of earlier 
achievement experiences and the current aspiration levels as well as 
some more stable personality characteristics which may be of import- 
ance in this connection (for example, what McClelland and others have 
called "need Achievement"). The fact that such complications do not 
require much attention in the world of rats and pigeons (and some pro- 
grammers always seem to think they will find the solutions to problems 
of the psychology of learning right in the rat laboratory), should not 
keep us from being aware of them in human learning situations. 

We should consider at least to aspects of the problem complex. On 
the one hand we should note the fact that the learning effectivity of an 
instruction material is often greater when there is a high frequency of 
positive feedback. This was one of the chief viewpoints of the early pro- 
grammer group inspired by Skinner, and it was usually assumed that the 
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error frequency should be low (i5crhaps under 5 %). On the other hand, 
the maintenance of motivation in the long run (consistence in work) also 
seems related to whether demands arc high encugh in relation to ability. 
Individuals with high "need Achievement" and good learning technique 
can, for example, find very boring a long series of all too easy tasks, 
and this, reasonably enough, may decrease their interest in efficiently 
working with the material. 

In this event we should perhaps try to make use of a " multiple success 
strategy ". This can mean that while we work fairly constantly with a 
high probability of correct answers to separate tasks, we combine that 
procedure with "extra success effects" - for example, the possibility 
of rapid-track progression and/or the satisfaction of being able to an- 
swer correctly with'^ut clues (individualized offering of clues as in 
teaching machine Didak 50Jl). Becker (1963) who proposes such a method 
of procedure compares this strategy with that which is used in certain 
lotteries or chewing gum machines. For one^s "contribution" (a coin) 
one always receives a certain "prize" (a piece of chewing gum ), but 
there is also the chance of getting a larger "prize" (an attractive object). 
A play of this kind on two planes of success should - mutatis mutandis - 
be serviceable in teaching material and might perhaps better do full 
justice to the varying individual needs. 

The reviser cannot, naturally, make too radical changes in the total 
structure (these should be allowed to wait until the empirical data also 
arc available). But he should make sure that nothing in the study 
material which can encourage boredom or discourage motivation is 
allowed to pass. Any possible monotony can ordinarily be cased through 
small changes in working methods. And if the reviser finds that the 
primary material provides all too little scope for the ambitious student 
to feel that he can achieve something on a par with his ability, then the 
addition of "extra success effects" should not be too difficult. 
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12. 5 VARIOUS AIDS IN THE CHECKING PHASE 

The fact that successive empirical try-outs of the rmterials are the 
final and most important steps in the evaluation process does not mean, 
of course, that we have to rush into premature data collection with 
hastily produced and unpolished versions of the study material. While 
an arbitrary reading through of the material may be fairly inefficient, 
a systematic mapping of certain apparent characteristics of the mate- 
rial, on the other hand, may be very fruitful. Wo may call this examina- 
tion of apparent characteristics , the mapping of the ' ‘ phono - structure ** 
of didactic sequences (as distinguished from the mapping of the “ effect - 
structure ** that might be carried cut during the empirical try-outs later 
on). 

In the present section several aids for this mapping process will be 
described and discussed, including check lists, diagrams of the rela- 
tions between terminal objectives and the single didactic units, and 
various kinds of unit -cha rting protocols. It is felt that the examination 
is considerably facilitated by summarizing aids of these kinds. The 
risk of forgetting some important aspects of the examination is reduced, 
and a long series of questions about the material can be answered much 
more easily on the basis of a summarizing protocol than on the basis of 
the non-aided perusal of a long sequence of material, in which the 
different parts cannot be kept in sight at the same time. 

It should be stated from the beginning that many different types of 
aids are possible, of course, and that each programmer should design 
his working tools to suit his own needs. The following descriptions 
should therefore be looked upon only as illustrations of various possi- 
bilities. 

12. 5. 1 Examination by Means of Check Lists 

The simplest way of remembering the important aspects that should be 
examined is probably to make up a fairly detailed list of questions to 
be answered for each self -instructional study material and to use this 
list for checking eff systematically the answers for each separate part of 
the course material. An example of such a list of questions is given in 
Box 12. 1, (It should perhaps be added that some of the questions in this 
list are such that only the subsequent empi rical txy -out can give a 
definite answer. Nevertheless, it is often possible to make intelligent 
guesses in advance, and thereby to save some steps in the revision 
process. ) 
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B OX 12, 1 Check list for pre-test examination: An illustration 



1, Examination of goal relevance and content: 

1, 1 Does the program contain all that it should according to the 
initial goal statements? 

1.2 Has all irrelevant material been eliminated? (Irrelevant mate - 
rials arc those that are neither included in the goal statements, 
nor fill any clear-cut educational function as aids in the process 
of reaching these goals, ) 

1. 3 Have all those points been eliminated that, according to the tar- 
get population analysis, the students have already mastered? 

1.4 Does the program avoid giving any single subject-matter aspect 
a coverage that, according to the goal statements, could be 
considered unreasonably large or unreasonably small? 

1. 5 Has all changeable information been checked, so that the 
students arc given maximally up-to-date information? 

1. 6 In the judgment of subject-matter experts, is the content free 

from factual errors ? 

2. Examination of the '^procedural instruction" of the program : 

2. 1 Arc the ’’procedural instructions" of the program to the student 
(instructions about the ways of handling the material during 
study etc, ) complete, clear and easy to follow? 

2, 2 Is the student given a sufficient amount of training in how to 

give answers or handle the teaching machine in the beginning of 
the program (as well as in other places whore perhaps a new 
answering technique is introduced)? 

3. Examination of the organization and sequence of content : 

3, 1 Docs the program try systematically to establish a connection 
between those experiences or concepts that the students already 
have in their repertoires and the new knowledge that is intro- 
duced ? 

3, 2 Is the program constructed to use - where this is possible and 
appropriate - the sequential approaches that are usually re- 
ferred to. by phrases such as "from the simple to the complex", 
"from the easy to the difficult", "from the concrete to the 
abstract"? 
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3. 3 Are there sufficient cxcrciecs for application and rc^Dctition? 

3.4 Arc the units with material for repetition adequately spaced 

(in the beginning sufficiently near the first presentation to make 
correct answers possible and after that with increasing and 
sufficient intervals to maintain retention over large enough 
periods of time)? 

4. Examination of the stimule-function of the single didules: 

4. 1 Have irrelevant and distracting details been removed, so that 
the attention of the student is clearly focused on the central 
part of the information of the didule? 

4. 2 Is the communication presented in a sufficiently life-like and 
concrete way? For instance, have pictures and supplementary 
demonstration materials been used where such aids can supply 
the intended information more effectively than verbal descrip- 
tion ? 

4, 3 Arc the examples given sufficiently varied so that the student 
docs not get a wrong and one-sided picture of the principle or 
concepts being treated? 

4.4 Is the position of information appropriate, so that, for instance, 
important information has not been placed after the response 
request ? 

4, 5 Have the prompting techniques used been sufficiently varied? 

4. 6 Has unnecessary prompting been avoided? 

4, 7 As a rule, has meaningful prompting (by means of logical in- 

duction, parallel exemplification etc. ) been chosen where 
appropriate rather than unnecessary formal prompting of a kind 
that may distract the student^s attention and divert it to unim- 
portant aspects of the communication? 

5. Examination of the rcspulc -function of the single didules: 

5. 1 As a rule, can the questions be answered on the basis of the in- 
formation that the student has received? 

5. 2 As a rule, have response requests been avoided that can b'^ 

answered without understanding of the content of the didule (by 
formal -linguistic pattern completion etc.)? 

5. 3 Has the response request been designed so as to give a reason- 
able guarantee that the student has understood the css ential 
communication of the particular didule? In other words: Are the 
response requests relevant to the aim of the particular didule 
(always assuming that each didule has a specifiable aim)? 
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5.4 Docs the program activate varying types of response behavior 
(writing, drawing, calculating, comparing, etc.)? 

5, 5 Do the alternatives of a multiple -choice question represent all 
reasonable, non-trivial sources of misunderstanding? In other 
words: Is it difficult to think of typical student errors not in- 
cluded among the alternatives listed? 

5. 6 Arc the ciltcrnativcs in a multiple -choic e question so designed 
that the stud eht^s choic c is not a p s eudo-.ch.ajLce (c. g. , so that the 
incorrect alternatives are not clearly absurd or formally in- 
congruent with the main question)? 

5. 7 Does the program request the student to give those ty^Des of 

answers that the terminal situation will demand from him (so 
that, for instance, the program does not merely train the student 
to recognize correct decisions, if in the terminal situation the 
student^s ability spontaneously to make correct decisions in also 
going to be tested)? 

6. Examination of the integrative functic., of the progr am; 

6, 1 Is the student given au opportunity of applying concepts and 

principles, first p resented stepwise, to problem-solving of a 
kind that forces him to activate and operate upon several con- 
cepts or principles at the same time? 

6. 2 Are integrating materials in the form of reference tables, re- 
ference diagrams, reference maps or reference texts utilized 
where appropriate? 

6, 3 Does the progr m help the student to obtain an adequate initial 
organization (”Gcj**alt in advance”, ’’properly structured ex- 
pectations”) as well as an adequate final organization ("final 
Gestalt”, e. g. , by means of "properly structured reviews”)? 

6.4 If a programmed textbook is used, does the student get a table of 
contents outlining the broad scope and structure of the total con- 
tents ? 

6. 5 Are there adequate indices that make it easy for the student to 

brush up his knowledge afterwards within specific areas of infor- 
mation, if ho so desires? 

7 . Examination of the motivating function of the program : 

7. 1 Have the programmers tried to provide a sufficient degree of 
variation in the review items? 
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7, 2 Docs the difficulty level of the program appear, throughout the 
program, to be so adapted to the particular studciit group under 
consideration that there is a fair probability that the students 
will be neither bored nor discouraged? 

7. 3 Have the examples and illustrations - to the extent that seems 

possible - been made interesting and relevant to the experiences 
and needs of the particular student groups? 

7, 4 Do the possibilities of individualization within tlie program 

(available branching arrangements) correspond closely enough 
to the results of the initial analyses of the target population, so 
that, for instance, students with considerable previous know- 
ledge of the subject matter arc not forced to go through the 
same material as students without any previous knowledge? 

8. Examination of the external form of the program: Language 

8, 1 Do the level of vocabulary and the structure of sentences seem 
to be sufficiently well adapted to the linguistic habits of the 
particular student group so as not to be an unnecessary barrier 
to communication of the main contents? 

8. 2 Is the meaning of new terms defined or otherwise demonstrated 
clearly enough as soon as they are introduced? 

8. 3 Js the language of the program always clear and unambiguous? 

8. 4 Docs the language always have an optimal degree of precision 
as judged from the goal statements of the program (that is, 
neither with too low a degree of precision, including sloppy 
everyday terms where precision is needed, nor with an un- 
necessarily high degree of precision, such as various scientific 
distinctions not to be further utilized in the desired terminal 
behavior) ? 

8. 5 Is the use of numbering, punctuation, abbreviation etc. both 

correct and consistent? 

9. Examination of the external form of the program: Other aspects 

9. 1 Is the general lay-out of the program both educationally appro- 
priate and economically defensible? 

9# 2 Is the typography sufficiently clear for the intended student group 
and also designed to emphasize points that shouJ.d be emphasized? 

9. 3 Arc illustrations of various kinds, both within the program proper 
and in possible appendices to be used as reference material, so 
designed technically that they can be expected to result in the 
best possible communication effect? 
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12. 5. 2 Examination by Means of Diagrams*’ Depicting the 

Relations between Terminal Objectives and Single Didactic Units 

Even during the writing process proper many programmers find that it 
makes the work easier to draw simple diagrams of the relations be- 
tween the terminal objectives as codified in the goal statements on the 
one hand and the single didactic units (the didules) as appearing in the 
written program on the other. Very often the program writer arranges 
both the specific goal statements and the single didule texts on separate 
cards in a card catalogue. This means that the diagrammatic survey 
often takes the form of a series of relational diagrams, from which it 
is easy to see which didule cards (D ^ etc, ) are related to each 
single terminal card (T^, etc, ), During the post-writing phase it 

is easy to examine, with the aid of diagrams of this kind, whether or 
not certain parts of the total goal structure have been given too much 
or too little attention in the program. Similarly, it can easily be seen 
whether or not enough attempts seem to have been made to roach an 
integrated structure of knowledge. Figure 12. 1 shows an example of 
such a diagram. The numbers of terminal objectives and didules in- 
cluded in the figure (T- and D -numbers) refer to the program fragment 
in Box 12, 2. 

If we deal with a so-called scrambled book, the diagram should in- 
clude not only information about the main part of any particular didule 
(the page with go-ahead signal and new information), but also informa- 
tion about the er ror -treatment pages (the pages that explain errors and 
refer the student back to the main pages again). As a rule, in these 
cases the programmer should also use a separate "pagination control 
list’* for checking off page numbers used. The reason is, of course, 
that the programmer and the post-writing examiner have to keep an eye 
on the scrambling system, sometimes fairly complicated, in order to 
ensure a suitable sequence of pages, that is, a sequence which is in- 
structionally adequate as well as economical with respect to time and 
space. (No pages should be left empty; the student should not need to 
turn too many pages each time; tasks that might malce the answer easier 
than intended or that would oterhwise interfere with each other should 
not be placed side by side on the same two -page fold -up of the book, 
etc, ) 
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Box 12, 2 Terminal descriptions (T cards) and didules (D cards) 
from a program fragment on programmed instruction 
(as supplemental rriaterial to the T-D diagram rn 
Figure 12. 1) 



Topic: Introductory course on programmed instruction 
Subjects: Teacher students 

Note; This fragmentary illustration contains 20 didules (D), 
covering 7 specific terminal objectives (T) - focusing, in turn, on 
3 fairly basic "principles”. 


T 1. 


The student should be able to state some acceptable reason 
for the emphasis on self -pacing in self -instructional ma- 
terials. 

(Principle 1) 


T 2, 


The student should be able to recognize and recall the term 
"self -pacing”. 

(Principle 1) 


T 3. 


The student should be able to state some acceptable reason 
1 for the emphasis on "overt responses" in self -instructional 
materials. 

(Principle 2) 


T 4. 


The student should be able to recognize and recall the terms 
"overt response” and "covert-response”, 

(Principle 2) 


T 5, 


Given a set of instructional items, the student should be able 
to differentiate between items demanding "overt" and "covert" 
responses. 

(Principle 2) 


T 6. 


The student should be able to state some acceptable reason 
for the emphasis on low error -rate in self -instructional 
mate- rials, 

(Principle 3) 


T 7. 


The student should be able to state some specific charac- 
teristics in self -instructional materials that make low error- 
rate possible, including, at least, the two notions of "small 
steps" and "careful seqviencing", 

(Principle 3) 




D 1. 


In the classroom, the teacher most often works with the total 
class. If so, all students have to go forward at the same 
speed - for instance, at the speed of the average child. This 
rate often tends to make the work (more/less) interesting to 
the bright and quick-working child. 

(Cf T 1) /less/ 
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D 2. 


What would you guess about the effect on the slow learners 
of following the average student^s speed? 

Probably, they (will/will not) grasp enough of the material 
presented in the time allowed. 

(Cf T 1) /will not/ 


D 3, 


If the slow learner does not grasp enough of the material 
presented, he will easily develop a (positive/neutral/negative) 
attitude towards classroom learning. 

(c.f T 1) /negativr>./ 


D 4, 


Students working at their own speed are said to follow the 
principle of "self -pacing". 

Presumably, the slow learner would be able to grasp more 
fully the material to be studied, if he were allowed to work 
at his own speed. Consequently, he would probably also 
develop a more positive attitude toward classroom learning, 
if the principle of were used, 

(Cf T 2) /self -pacing/ 


D 5, 


vSince the quick learner is less likely to lose interest in the 
work (less likely to get bored), when allowed to proceed as 
quickly as he is able to, the principle of also 

favors the bright and quick pupils. 

(Cf T 2) /self -pacing/ 


D 6, 


To sum up, then, there is reason to believe that "teacher- 
pacing" is (more/less) likely to be a favorable learning con- 
dition than " ", 

(Cf T 1 & T 2) /less; self -pacing/ 


D 7. 


The student who actually works through a statistical calculation 
is (more/less) likely to learn effectively than a student who 
merely reads a description of what to do. 

(Cf T 3) /more/ 


D 8. 


When a student learns, he may make both overt and covert 
responses. Those activities we can easily observe (like - 
writing or manipulating a machine) we call overt response. 
Those activities we cannot observe (like thinking) we call 

responses. 

(Cf T 4) /covert/ 


D 9. 


Using our technical terms "overt" and "covert" responses, 
we may say that the student of statistics who merely reads a 
discription of what to do, makes responses 

to the text, where as the student who works out the calculation 
in writing in addition shows responses. 

(Cf T 4) /covert; overt) 
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DIO, A house -wife who tries to learn cooking from a book just by 

reading may be said to make only responses 

to the text. There is reason to believe that this is a (more/ 
less) effective way of learning cooking than actual practice, 

(Cf T 3 &: T 4) /covert; less/ 



Dll, (Observable/Non-observable) reactions arc, of course, a 

better guarantee that the student has actively responded to all 
the important aspects of the materials than arc (ovcrt/covert) 
reactions , 

(Cf T 3 T 4) /observable; covert/ 



D 12, To sum up, then, covert responses are (more/less) likely to 
be effective in learning than arc responses, 

(Cf T 3 & T 4) /les s; overt 



D 13, Look back to the three sentences of item 8 above. In which 
sentence were you c::pcctcd to make an overt response? 
Answer: In the sentence. 



(Cf T 5) 



/ thi r d / 



D 14, Look at the four items in Panel I, Put checkmarks in the list 
below to indicate the kind of response expected in each case 
from the student. 



Item 


Covert only 


Overt 


A 






B 






C 






D 







(Cf T 5) /Note, Panel I is excluded from the present 
illustration/ 



D 15, A student who is anxious or uninterested in the work may not 
be motivated to continue working, A student who makes very 
many errors may get anxious or uninterested in the work, A 
student who makes very many errors, therefore^ (is/is not) 
likely to be motivated to keep on working, 

(Cf T 6) /is not/ 



D 16, If we make an error, we tend to remember the error, if it is 
not corrected. If we give a correct answer, we tend to re- 
member the correct answer. In the first case we have to '*un- 
Icarn" the error. In the second case we ’learn" directly. Con- 
sequently, learning might become more efficient, if we could 
arrange the study material so that the students made (many/ 
few) errors, 

(Cf T 6) /few/ 
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D 17. If the student is going to reach the goal we have set without 
making many errors, we have to arrange the study material 
in (small/large) steps. 

(Cf T 7) /small/ 

D 1 8. If ITEM X is something the student needs to know before he 

can learn ITEM A and ITEM B, we should begin his sequence 
of items with ITEM . 

(Cf T 7) /X/ 



D 19. In order to make sure that the student makes few errors 
only, it would seem reasonable to recommend the use of 
steps and a careful of items. 

(Cf T 7) /small; sequence/ 



D 20. To sum up, if wc use small steps and a careful sequence of 

items, wc arc likely to get (more/fevver) errors. In that case 
motivation to keep on working will usually be (highcr/lowcr) 
and the learning will proceed (more/less) efficiently. 

(Cf T 6 T 7) /fewer; higher; more/ 
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12. 5* 3 Examination by Means of Didulc Protocols 

A didule protocol is a device which makes it easier to make the examina- 
tion both complete and systematic. Examination by means of general 
checklists aids the examiner in covering many aspects, but it may not 
by itself help him to cover every single part of the program i7i a 
systematic way. As a rule didulc protocol includes, for each didule, a 
separate examination column, in which - depending on the particular 
aim - various kinds of questions maybe answered when this particular 
didulc is examined. Three general types of such protocols may perhaps 
be distinguished: subject-matter protocols, methodological protocols, 
and combined protocols studying both subject-matter distribution and 
the methodological approaches used, 

12. 5, 3, 1 Subject-Matter Protocols 

The main aim of the subject-matter protocol is to some extent similar 
to the aim of the T -D diagrams described above, viz. to facilitate the 
examination of the distribution of the subject-matter over the didulc 
sequence, A square-ruled paper may be a good starting -point. The 
numbers of the didulcs arc written in numerical order along the top 
of the paper. The various subject matter units that are to be examined 
arc listed down the left-hand m^argin. For instance, these units may be 
single conceptual units or conceptual relations which have been listed 
during the pre-writing phase as important key points to be covered in 
the program. Sometimes it is more natural to use psychological units 
than logica) units, that is, to start out from a series of stimulus- 
response connections rather than from concepts and conceptual rela- 
tions, Whatever the particular kind of units used, however, the general 
examination process will be similar. By means of some kind of check 
marks in the appropriate cells, we report in which of the didulcs a 
particular subject-matter unit is treated (cf. Figure 12. 2). 

Among the questions that may be answered with the aid of such a 
protocol, the following maybe mentioned: (a) Is each single basic 
subject-matter unit represented to a sufficient degree in the program? 

(b) Arc some of the didules totally irrelevant to the main subject-matter 
to be covered? (c) Is the treatment of the single subject-matter units 
appropriately distributed (for instance, an intensive scries of didulcs 
at the first p res entation, followed later on by separate repetition items 
with gradually increasing intervals)? (d) To what extent have attempts 
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been made to integrate the separate units into meaningful structures, 



instance, where the separate concepts are sufficiently well treated 
and practised)? 

When answering the last question, for instance, the examiner will 
study to what extent the same didule column contains check marks re- 
ferring to several different subject-matter units. It is usually desirable 
that this is the case in many didules in order to facilitate integrated 
structures of knowledge. At the same time the examiner of the pro- 
gram studies at what specific points in the didule sequence the new 
subject-matter units are first presented. As a rule it is confusing to 
the student, if several new subject-matter units arc introduced in the 
same didule. An integrating didule therefore usually contains either a 
scries of units introduced earlier in the program or one new unit of 
subject-matter in combination with one or more units introduced 
earlier. 

As mentioned above it is sometimes maintained that integration of 
knowledge may be facilitated by specific patterns of didules, such as 
systematic alternation ("counterpomt"), systematic overlapping, 
chronological accumulation, or reversed accumulation of temporal 
behavior sequences (cf. section 12. 2). Further research is needed 
on these and other patterns of didules recommended for integration 
purposes. In this experimentation . it is, of course, valuable to be 
able to examine, by means of subject-matter protocols, the ax^pcarance 
of the various patterns under study. It is then easy for the examiner to 
observe inadvertencies and to correct arrangements which have been 
left in a less than optirral shape during the writing process. ( A few 
examples of typical patterns referred to by t ie terms used above are 
given in Figure 12. 3. ) 

12, 5. 3. 2 Methodological Protocols 

The purpose of the methodological protocols is to facilitate the examina-, 
tion of various methodological approaches and make it possible to an- 
swer systematically questions of 'he following typos: 

(a) Has the programmer got stuck in certain working routines with the 
accompanying risk that the student finds the program monotonous? 

(b) Are the educational techniques used of such a kind that, besides 
facilitating the student'^s learning of certain subject-matter facts, they 
also contribute to educationally desirable side-effects (such as good 
working habits, positive study attitudes, etc. )? 



and arc these attempts rr.adc at optimal points in the sequence (for 




Figure 12> 2 An example of a subject-matter protocol (didule protocol, type A) 
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Integra -ion with other subject-matter units. 
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Figur e 12,3 Examples of special tyjDes of diclule sequences as 
reflected in a subject-matter protocol 



Note: In this figure, numbers have been given to subject-matter 

units that make up a connected temporal chain, whereas lette rs 
have been given to subject-matter units that are not temporally 
related to each other. A small arrow below the didule number 
shows in which order different parts appear within a single didule. 



a. Systematic alternation ("counterpoint** ) 
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c. Chronological accumulation 
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The protocol sheets will be similar to the ones used for subject- 
matter protocols, that is, square -ruled papers with didule numbers 
along the top. However, questions about the educational approach or 
key-terms for different methodological categories (instead of subject- 
matter units) are now written down the left-hand margin. If the question 
is answered by ”yes** for a certain didule, the examiner puts a check 
mark in the appropriate cell (where the didule column and the row of 
the question intersect). Similarly, he inserts a chejk mark whenever 
the didule under study can be classified under the methodological cate- 
gory listed in the margin. 

A very simple question that may be included for study in a protocol 
of this type is the following: Where has the student to go in order find 
a solution to the task set for him in the iidule? Among the various 
possible sub -categories representing different groups of conceivable 
answers to this question, we may, for instance, mention the following: 

(1) The solution can be found directly in the text of the same didule 
('Unternal-explicit'* source). 

(2) The solution can be indirectly deduced (through a process of 
drawing conclusions etc. ) from the new information that is given in 
the same didule ('^internal -implicit*’ source). 

(3) The solution can be found in the information presented earlier in 
the program ("external -proximal" source). 

(4) The solution cannot be found v^ithin the program, but can be con- 
structed in some way, usually with the aid of the student's back- 
ground experience and/or his ability to use logical reasoning (or 
mathematical operations) ("external-distal** source). 

At first sight, a subdivision of this type may seem to represe nt an 
unnec es sary and pedantic eagerne ss for box-sorting. However, the 
observation of "response sources" is a very important part of the didule 
examination, and many existing programs would probably have looked 
quite different and been more effective, if the programmer had been 
made aware early enough of one-sided techniques utilized. The special 
terminological labels used ("internal, explicit*’ etc. ) should not, of 
course, be taken too seriously, but a brief and handy term may be use- 
ful in writing out these protocols as well as in discussing these pheno- 
mena. 

Some of the categories mentioned above can be further subdivided, 
if desirable. The third category, for instance, may be divided into 
maximally proximal cases (when the answer is to be found in the 
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immediately preceding didule) and other cases (when the answer is 
found earlier in the program). Separate "appcndices**(or ’'panels*') as 
response sources can, when appropriate, be listed as a separate cate- 
gory, etc. 

Such simple mapping gives the examiner quick information about the 
possible monotony of a sequence, showing, for instance, whether or 
not the programmer has used exclusively ’’internal-explicit" sources 
(which seems very often to bo the case in some published programs). 
Each technique may serve an important function. The internal -explicit 
approach focuses the attention of the student on the key points in the 
immediate text and seems to be of particular value when new terms 
and concepts are introduced. The external -proximal method contributes 
to the gradual training of retention and may also be used in the process 
of integration. In order to ensure valuable educational side -effects 
(good working habits etc. ), the programmer will probably often find it 
most profitable to use either an inte rnal -implicit or an external -distal 
technique. 

Other questions that can be studied in a similar way, arc: What 
type of activity is expected from the student (for instance, copying, 
guessing, analogous exemplification, induction from own experiences, 
logical conclusions)? What kind of prompting is used? How is the an- 
swer related to earlier answers and to the key -points of the didule? 

What is the linguistic structure of the didule? (For instance, how high 
is the "density" of information-loaded words ? ) Etc. etc, (For further 
illustrations, cf. Figure 12,4,) 

In certain cases it might be desirable to calculate some index score 
covering the "educational variation" (the distribution of didules with 
respect to a series of sub -categories related to a certain methodological 
question), or some index score covering the proportion of didules in- 
cluding certain educational features probably rcsiilting in desirable 
side effects (when some of the sub -c ategories belonging to a certain 
methodological question are thought to be educationally more desirable 
than the other sub-groups). Such calculations should not, of course, be 
used too routinely, nor with too great a confidence in the exactness of 
the scores derives. Nevertheless, they can serve an important function 
as general warning signals, telling the examiner, in fact, "The program 
ought not to look like this. Revision urgently neededP’ The methodolog- 
ical protocol can then show more exactly which particular points 
should be revised. 
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Figure 1 2» 4 An example of a methodological protocol (didule protocol, 
type B) 





Didule s 


Methodology: 


1 


2 


3 |4 


5 


h iM 


1 8 i 9 1 10 


1. Response sources: 




1. 1 internal-explicit 


/ \ 


\/i 

A\l 




V / 




X 


XT — 
XX i 






1. 2 internal -implicit 




1 

i 


i 


1 














1. 3 external -proximal 




— 
















X 


1.4 external -distal 








1 














2. Types of prompting: 
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2. 4 parallel exemplification 
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Reading key: In didule 1 the student can obtain the information needed for 
his answer directly from what is said within this didule (*'internal-explicit*' 
response source), and the answer is facilitated by strong formal prompting 
(word -.form prompts, emphasizing the key words, and syntactic controls). 
Etc. 



Comments: The example shows a one-sided use of ^internal-explicit^' response 
sources as well as formal prompting. In two cases, however, there is no 
prompting at all, but the student is expected to recall material from earlier 
didules (no. 6 and no. 10). In no case is use made of parallel exemplification, 
induction from experience, or logical conclusion. If the one-sidedness 
illustrated here continues over a larger material, there is the risk of both 
monotony and less adequate side-effects (lack of training in appropriate study 
techniques). 
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IZ. 5, 3. 3 Combined Protocols 

Combined protocols register both subject-matter distribution and par- 
ticular methodological approaches in some kind of combination. In 
many cases it is not enough to see whether the subject-matter is 
suitably divided over the didules, and whether the methods used are 
sufficiently varied. We also want to make sure that the various methods 
enter at the most appropriate points in the subject-matter sequence. 
Sometimes we consider certain methodological approaches more 
suitable when a new concept is introduced than when the concept is al- 
ready relatively well incorporated in the behavior repertoire of the 
student. The types of prompting techniques that we are perha.ps reluctant 
to accept whe 1 wc deal with a terminology that is well known to the 
student (for instance, prompting of a wholly formal -mechanistic type), 
we may sometimes find acceptable, or even natural, when we introduce 
a new term. In addition, wc sometimes want to make sure that wc do 
not connect a certain subject-matter area too closely with a certain 
method in the student^s behavioral repertoire. Not only do wc want to 
have an overall methodological variation, but now and then wc also 
have to check that we use a methodologically varied technique in the 
treatment of each single subject-matter area. 

In some respects, the combined protocols are very similar to the 
subject-matter protocols: a square -ruled arrangement wi^h didule num- 
bers along the top and subjcct-ir^atter units listed down the left-hand 
margin. However, instead of the simple check-marking process, wc 
now use a scries of different symbols for various methodological cate- 
gories (cf. Figure IZ. 5). 

IZ, 5, 4 A Final Note 

It should be mentioned that protocols and diagrams of these types may 
often be good training instruments for persons wanting to train their 
ability to observe educationally important differences between various 
published programs. Hence, in courses about self -instructional materials 
and programming, protocols of these and similar types seem to be 
well suited to make the practical training of the study group more 
effective and efficient. However, the main aim of these protocols is, 
of course, to be an aid in the process of constructing a program making 
the programmer or an independent examiner aware of possible faults 
in the program even before the first empirical try-out. As stated above, 
it is hoped that this will reduce the work and cost involved in the final 
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Figure 12^ 5 An example of a combined protocol (didule protocol, 
type C) 
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Reading key: 

Symbols for response sources: 

o = internal-explicit 
=r internal-implicit 

s external-proximal 
external -distal 



Symbols for prompting types: 

W t: -word-form prompts 
M formal emphasis 

I = induction from experience 
=: logical conclusion 
E parallel exemplification 



The protocol is read in the following way: The s\ibject matter unit C% is treated 
in the didules 2, 3, 4 and later on also in numbers 8, 11 and 15« In aidules 
Z and 3 the student obtains his ansv/er within the same didule (first with the 
aid of word-form prompts, then with the aid of formal emphasis). In the other 
cases with Cj the student has to rely cn memory* Etc. 

Comment; 

The didule sequence seems to be well constructed* The micro sequences go 
from fairly easy tasks (with prompting) to more difficult ones (without prompting), 
non-formal prompting is frequently used, and the sequences contain problems 
that contribute to the integration of knowledge as final elements (number 8, 11 
and 15). 
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As our scientific knowledge of the self-instructional learning process 
is gradually increased, the questions studied by means of these check 
lists, diagrams and protocols will be more specific, and the evaluation 
will be more securely founded on hard facts. This will probably mean 
that the careful study of ”phcno-structures*' by means of aids of these 
kinds will gradually become more important and more economical. 

The empirical try-outs with subsequent revisions will probably always 
remain an important final control process, but the amount of work in- 
volved in it will decrease, and at the same time it will be possible and 
natural to pay increased attention to the post -writing, pre-testing exa- 
mination. 
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13. THE PROCEDURE OF SUCCESSIVE EVALUATIONS 
13. 1 INTRODUCTORY VIEWPOINTS 

A comparison between different types of instruction (for example, 
lecture, conventional classroom instruction, group work, private 
tutoring, and programmed instruction) using a systematic examina- 
tion of various characteristics (for instance, presence or absence 
of disturbance effects, tlie degree of data storage, and the degree of 
feedback from student system to teacher system) seems to give se- 
veral points of departure for a discussion of educational effective- 
ness (cf. Figure 13. 1). Especially notable seems to be that the pro- 
grammed instruction is the only situation which combines (1) con- 
tinous feedback from student system to educator system with (2) 
complete data storage. 

Among other things this means that the response residuals in the 
used-up programs give good starting-points for revisions (cf. also 
Figure 13.2). Usually, therefore, the construction of self- instructional 
material is a pragmatic procedure in which one considers and acts 
upon the feedback information., That which functions well is retained, 
but that which does not function satisfactorily is taken away or changed. 
Another way of stating the basic characteristic of this way of acting 
in the system control phase: A program is not really a program until 
it works. Or even more drastically: One should not be surprised that 
programmed instruction works. It works per definitionem ! 

The examination of the actual interaction between student and pro- 
gram is therefore of basic importance. This means, on a small scale, 
a Copernican revolution in the way of looking at student achievement. 
Traditionally many people look at student achievement in the first place 
as a measure of student ability, cUid those students who do not seem 
to achieve much are sorted out from the system of the school. In the 
evaluation of self-instructional materials, on the other hand, student 
achievement is considered primarily as a measure of the effectiveness 
of the teaching materials, and instructional units which do not seem to 
lead to good student achievement are sorted out or replaced. (Of course, 
probably nobody believes that all students are able to learn everything. 
But a program evaluation with successive revisions based on feedback 
information mecUis a serious attempt to adapt the method to the individual, 
thereby trying to reach more individuals in a more effective way.) Pro- 
grammed instruction has been one of the major influences behind the 
increased interest in what is now often called mastery learning (cf. 

Bloom, 1971). 
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Fijgure 13. 1 The role of feedback and data storage in various types 
of instructional s\^stems 
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1) Quantitative values for what is hereby to be considered “continuous” 
“partial”, or “rudimentary” can hardly be given, but most judges 
will still be able to agree on the classification of the typical cases 
over this rough scale. 
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^ • *’Group work** 

( Partial feedback from e due and 
to educator ; rudimentary ^^data 
storage via the educator system; 
disturbance effects, but also 
partial educator effects between 
educand systems) 

1) In certain forms of group work 
with succes.iive work reports, 
the term "partial data storage" 
may be ihore adequate. 
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Figure 13.2 The three main phases in the interaction between student 
and program and three types of feedback effects 
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Three feedback effects: 

(1) "individual effect": addition to or change of the experience residuals 
of the individual, which means that the initial repertoire is different 
when confronted with a new educator system 

(2) "‘feedback to the program": in adaptive educator systems the infor- 
mation on the individual's responses influence the program's con- 
tribu.*'ion to the next phase of the interaction 

(3) "feedback to the programmer" : .the response residuals in the used- 
up program give indications of where and/or how to revise, that is 
lead to reconstruction in the preparation phase. 
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13.2 THE MAIN STAGES OF THE EMPIRICAL EVALUATION 

Revision work and the final testing can of course be carried out in many 
different ways. The main principle is an alternating sequence of em- 
pirical tests and revisions. The following arc some of the most usual 
work stages. 

13.2.1 First Phase: Explorative Tasting of Program Sections on 
Individual Students 

The first testing stage is usually considered one of the most exciting 
moments in programming. The program is tried out on individual stu- 
dents, It is one of those moments when the programmer's self-esteem 
can receive the most unpleasant jolts, namely, if it is shown that 
working sequences, on which he has devoted a great deal of intensive 
work and which he himself believes to be well night perfect, do not in 
fact work at all. In order not to be kept in suspense for too long many 
programmers consider it appropriate not to keep back this explorative 
and individual try-out until ti e whole program is ready. Instead, it 
may prove good strategy to test out small sections during the actual 
work of composition, before going any further. 

The students selected for this first try-out should, from a knowledge 
and intelligence point of view, be fairly typical of the group to whom 
the program will subsequently be directed, but preferably they should 
not be too docile and timid. Students who arc not afraid to criticize 
and proclaim their opinion are those who will prove most valuable in 
this first test stage. The intention here is not that the material should 
necessarily be gone through under '*ficld conditions". The most impor- 
tant aim is to get as detailed an idea as possible of the student's way 
of reacting to the details of the program. For this reason the try-out 
will preferably take place with one student at a time. As a preliminary, 
one ought - possibly witli special tests - to make certain that the stu- 
dent does not know the material in advance. One should also make clear 
to the student that the try-out applies to the study material not to him. 
Such a frank declaration usually creates favorable conditions for free 
and untrammelled communication without unnecessary anxiety. A quiet 
testing room and plenty of time arc also recommended. 

The first try-out is often made orally with student comments for 
each didule and with a detailed follow-up interview. However much the 
programmer values his own program, he must take care not to be on 
the defensive during these interviews. His task is not to defend himsolf. 
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but to collect reaction data. Should we begin to argue, we risk the stu- 
dent becoming unwilling to express his points of view. If we are dealing 
with adult students, the comments can sometimes be given in writing* 
The student, for instance, is given the instructional units in the shape 
of a collection of cards, and he writes his responses on a response- 
sheet, on which there is also space for marginal side-comments 
(e. g. , a sheet with two columns, one for responses and one for notes). 
Moreover, tlie student can possibly fill in a special questionnaire. As 
soon as the instruction has been tested with one or more students, the 
first revisions can be made. Ambiguities arc removed, better examples 
are chosen, illustrations arc made clearer, extra units are added, and 
so forth. At this stage the number of students is less important than 
the detailed analysis of how well the communication ‘'penetrates”. A 
small number of carefull; studied students is often a greater value than 
a larger number whose reactions arc more superficially studied. 

13. 2. Z Second Phase: Successive Testing of the Entire Program on 
Ltargcr Groups of Students 

When we consider that most of the obvious obstacles have been cleared 
away, the testing of the total program is begun under normal educatio- 
nal conditions (and with presentation aids intended for use in the field). 

A sufficiently large group is used so that a certain amount of fairly 
reliable information can be obtained concerning, for instance, the per- 
centage of mistakes in the individual instructional units. Pre- and 
posttests of knowledge are carried out in order to assess the total 
learning effect. The program is revised on the basis of analysis of mis- 
takes within the program and the terminal effects. Individual units arc 
discarded, additional tasks are inserted, the sequence of units is al- 
tered. Perhaps it is shown that extra branchings are necessary. After 
revision a new try-out on a new group of students is made. The pro 
procedure can be repeated several times. In principle, the group try- 
outs should be continued, until those values of error data and, above 
all, of terminal performance data have been attained that are considered 
desirable and agree with the objectives of the course. In order to fore- 
stall confusion and, in addition, in order not to give a false impression 
of having a fully adequate material, the program versions, which arc 
used in these preliminary try-outs, should have suitably unambiguous 
headings such as "Experimental edition 2”, "Experimental edition 3" 
or something similar. (A more detailed discussion of evaluation criteria 
will be given below* ) 
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13.2.3 Tliird Phase: T -rminal Field Testing 

When revisions arc no longer considered necessary, the timv^ is ripe 
for the final field tc? sting. In this the demands on the representativeness 
of the student groups and on the "normality" of the field conditions arc 
more severe. Exhaustive data on pro -knowledge, intelligence, time 
spent and final results are collected. Pre -knowledge and final results 
arc often measured best by parallel tests. In view of the student be- 
ing able to learn something also from the pre-test, and taking into 
consideration that the student can learn a groat deal from the study ma- 
terial without on that account attaining the same level of knowledge as 
a student taught by another material, it is often desirable to use the 
pre- and post-tests, for the sake of comparison, also in a control 
group (with a corresponding student group, but taught in a different 
manner). Another reason to use a control group for the pre-test is 
that this test can have a certain motivating effect, in that the student 
gains advanc . information about his own future possibilities after the 
end of the course. (A comparison between the first and the third testing 
stages is given in Box 13. i; an example of the planning schedule is 
shown in Box 13. 2. ) 

After this field test no alteration of the program should occur. 

Data of this testa's conditions and results will, namely, constitute an 
essential part of the basic information about the program^’s character 
for teachers and administrators. This information should normally be 
published in the form of a special "program manual", which can be 
said to be the program producer'^s "declaration of ingredients" for po- 
tential program consumers. 
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B oy. 13.1 A comparison between the first and third main stage in the 
successive try-out work 





First Stage 


Third Stage 


Aims 


Explorative tests of the 


Collection of data for 




ability of the program 


final ’’declaration of in- 




to communicate and 


gredients” for program 




teach 


consumers 


Time 


Several times during 


Not until entire program 




developmental work 


is in terminal state 


Main 


Program constructor 


Expert of investigation 


responsibi- 




Methods in behavioral 


lity 




sciences 


Te St 


A small number of 


A larger number of per- 


persons 


persons typical of the 


sons, an unbiased 




target population 


sample of tho target 
population 


Test con- 


Fairly informal with 


Strictly r eproduceable 


ditions 


good prospects for re- 


conditions, demands for 




laxed communication 


detailed informa'cion re 




and intensive observa- 


student characteristics. 




tion 


demcuids for the situation 
to correspond to the con- 
ditions under which the 
program will be used 


Follow-up 


Revision of program 


Tabulation and analysis 


measures 




of data re programme 
effect; no revision 
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Box 131 Z Planning and realization schedule: An example 

Note: Preparation, construction and follow-up work demand exhaustive 
planning. Here a planning Bchedule can be of great use. It should- in 
addition to the designations for the different work stages - include the 
names of those responsible for the different work stages as well as 
the date when the stagers work should be finished (PT = planned time), 
and when it actually was finished (AT = actual time). The latter de- 
tails can give a sounder ground-work for the planning of the next pro- 
gramming project. 




Project Project leader 


Task: 


Responsible; 


PT 


AT 


1. Main direction of goal 
analysis is established 








Z, Goal analysis, inclu- 
ding detailed terminal 
descriptions and ter- 
minal tests 








3. Student group identi- 
fied and analyzed 








4. Enlarged terminal 
chart and modified 
tests 




1 




5 . Situation analy si s 








6. Subject-matter ana- 
lysis: logical struc- 
turing 








7< Subject-matter ana- 
lysis: psychological 
supplementation 








8. Choice of presentation 
media 








9. Points 1-8 are co- 
ordinated and tabu- 
lated in the form of 
detailed directives 
to the program author, 
supplemented with T- 
catalogue, C -catalogue^ 
etc. 
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Task: 


Responsible: 


PT 


! 

AT 


10. Examination of al- 
ready existing didac- 
tic material. Possible 
decision on project 
alteration. 








11. Preparatory sequen- 
tial decisions: Choice 
of *‘flood model” 








12. Preparatory subject- 
matter structuring, 
possibly in the form 
of a didactoplan 




1 


1 

i 

( 


13. Version 1 is com- 
pleted 


i 

i i 

1 . j 


1 




14. Version 1 is re-eyamined 


; 1 
I 






15. Version 1 is revised 


i 


’ 


16. Version 2 is subjected 
to preiiiminary try-out 
with mdividual obser- 
vations 


1 

I 

1 

j 

j 






17. Version 2 is revised 








18. Decision on further pre- 
liminary try-outs 








19. Version x is group- 
tested 






^ ^ -J. 


20^ Version x is revised 








21. Decision on further group 
tests for revision pur- 
poses 






i 

1 1 
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Task: 


Responsible: 


PT 


AT 


22. Editorial final examina- 
tion 








23. Printing (or the like) 
of final v'ersion 








24. Field testing 








25. Writing of program 

manual with data from 
field testing 
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14. EVALUATION CRITERIA AND EVALUATION AIDS: SOME 
EXAMPLES OF RELEVANT VARIABLES AND SYSTEMATIC 
MAPPING DEVICES 

14. 1 INTRODUCTORY POINTS OF VIEW: DIFFERENT TYPES OF 
EDUCATIONAL MEASUREMENT 

In general, educational measurements can be made of conditions before, 
during, and/or after a particular inst tctional sequer ce, and they can 
aim at an evaluation both of the individual and of tne instructional s ystem 
(as is ou'..Iined in Figure 14. 2). 

In the past, work on educational measurement has above all dealt 
with problems concerning cell 3 in our figure: how one constructs 
measurement instruments that give as reliable and valid a picture as 
possible of the final competence of the individual, mostly in com- 
parison to some *^norm group'^ This is, of course, an important as- 
pect. However, other measurement aspects have by comparison been 
underdeveloped. First, it should be of interest not only to assess the 
status of the individual but also to evaluate the instructional system 
he is confronted with. How good are our systems (constructed to bring 
forth an optimal sequence of behavioral changes in accordance with 
given goal descriptions among the largest possible number of relevant 
students)? Second, it can well be said that so far we have been too 
often satisfied to obtain evaluations afterwards only (cell 3 and 6), in 
which case we primarily get comparisons between individuals or com- 
parisons between systems, while a combination of measurements 
during and after the instruction (2+3 or 5+6) would often give more 
"useful** information. By useful information we then mean such data 
that directly serve to localize weak links in the instructional, system: 
single parts that should be improved. 

Programmed instruction is one of the fields of recent educational 
interest that have served to turn the interest of educational measure- 
ment specialists in new directions and broaden the focus. When succes- 
sive revisions of the material are made, one is guided both by the be- 
havior of the students in the instructional situation (to what extent the 
student succeeds and how he reacts to single didules) and by their ter- 
minal achievements (their ability to reach the goals as measured by 
terminal tests). 
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Focusing on 
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Studying 
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of defects in 


total effectiv- 




between actual 


function; 
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student level 


control of 


efficiency of 




and difficulty 
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system in re - 




level of the 


branching 


lation to goal 




system 


system 


catalogue 



Figure 14. 1 General examples of various types of educational 
measurements 
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14. 2 TERMINAL VARIABLES: FINAJL ACHIEVEMENTS 

The terminal achievements are naturally enough the most important 
indicators. If we get good final results, then we need not bother so 
much about the previous behavior during the instructional situation. 
However, should we obtain a less satisfactory final achievement, then 
this is an important indicator that something is not in order, but this 
piece of information usually gives the programmer rather little specific 
guidance on where and how improvements should be made. In this case, 
then, study of the didule behavior (correct and incorrect reactions on 
single instructional units) is far more informative. 

The character of the final tests is obviously very important. If the 
test is unsatisfactory (unreliable or not representative as far as the 
goal of the course is concerned), then we are severely misguieded when 
we try to assess the effectivity of oui study material. The final tests 
should therefore always be worked out in close cooperation with experts 
in psychological and educational testing and with reasonable attention paid 
to usual measurement criteria (including demonstrations of reliability 
and validity). 

Since there are many books on these topics, we will not enter into 
any description of these general rules here, but refer any reader who 
feels that he does not know enough about these matters to common text- 
books on educational measurements. 

Besides these general criteria, we would like to emphasize three 
desirable characteristics of our measurement instruments. They should: 

(a) be so designed and so related to the specific instructirnal system 
that the results obtained in using them will have clear and meaningful 
consequences for further planning; 

(b) have a broad coverage, rather than be one-sidedly perfect; 

(c) be clearly goal-related, rather than content -related or norm- 
related. 

The first condition mentioned was already touched upon above in our 
brief discussion on Figure 14. 1, and we will return to concrete examples 
of this below. The second point may seem self-evident, but neverthe- 
less is sometimes too easily forgotten. One should keep in mind the 
total goal area of a course, so that one does not rest satisfied with too 
narrow a test instrument. The temptation to do this often derives from 
the fact that certain behaviors are more easily tested than others. If, 
for example, retention over longer periods of time or a certain kind of 
transfer effect are important parts of the goal area, one should try to 
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assess these things, even though they may be difficult to measure. 

Our measurement techniques are comparatively advanced as long 
as we deal w;ith the cognitive field. Nevertheless, there is a risk that 
we rest satisfied with tests dealing primarily with simple recall of 
separate items of knowledge (since such tests arc so easy to construct), 
in spite of the fact that we really want to cover more complicated abi- 
lities (such as application, analysis, or synthesis of knowledge items). 

In order to help us remember to cover those aspects that we intend to 
cover it may often be worth while to use test construction matrix of 
the kind presented in Figure 14. 2. In the headings of the columns the 
various cognitive functions we want to deal with are entered (for example, 
using the terms of the well-known taxonomy of the Bloom group; Bloom, 
1956), while the rows represent the various subject-matter areas. 

Such a simple cell system may be a good aid for the test constructor in 
not forgetting any important type of test item. (Cf. , for example, Payne, 
1968 . ) This does not mean, of course, that it is always easy to con- 
struct items which correspond to all the cells of such a matrix. We may 
find it especially difficult when we are interested in including not only 
cognitive, but also non -cognitive functions. In spite of attempts to con- 
struct taxonomies in non-cognitive fields also (see, e, g. Krathwoh.l, 

1964), we still have very few good test instruments available in these 
areas. 

In our third and last point on desirable characteristics we touched 
upon the frame of reference of the evaluation or the critcra used for 
comparison when we make our assessment. We may distinguish three 
main categories of such assessments; 

(a) Goal -oriented assessments . The assessment of the instructional 
result in then related to a specific series of goal descriptions. (** Out of 
the available test items, designed to cover the demands of our goal 
descriptions, Student 3^^ has been able to pass 90 %. '‘) 

(t>) Proc e s s -oriented (or content-related) assessments. The assess- 
ment of the instructional result is related to some specification of what 
has been presented during a course. ("Out cf tr ^ available test items, 
designed to be a representative sample of the content of the course - 
as this is seen from a particular book or summary of lectures, Student 
has been able to pass 89 %. ") 

(c) Product-group oriented (or norm -related) assessments. The 
assessment of the instructional result is related to information about 
results of other individuals. ("On the tests used Student So kas obtained 
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Figure 14, Z Test construction matrix as an aid in avoiding one-sidedness in item choice 
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a result which places him in the best 25 %. ”) 

All three methods can give valuable information in some situations - 
and sometimes, in addition, rather similar information. In other cases, 
however, the similarities arc less marked, and then it is reasonable 
to ask which of the methods is most meaningful for onc^s particular aim. 
It might first be noted that the third type of assessment is most clearly 
different from the other two in being more relative in character. Thus, 
we may talk about ’’absolute” or ’’criterion-related” assessment on the 
one hand (and to this group we may then refer both the a and the b type) 
vs. ’’relative” or ’’norm -related” assessments on the other (the £ type; 
cf. for example the dichotomy in Glaser, 1963). 

Earlier educational measurement was usually dominated by norm- 
relatcd measurements. Then the main aim was to differentiate among 
individuals, for example, in order to make individual predictions of 
future study success. Test results arc thus mainly seen as prognostic 
instruments (we predict from cell 3 - cf. Figure 14. 1 - at time 1 to 
cell 3 at time 2 etc. ). But for many other purposes, especially when 
we arc interested in the effectiveness of a particular instructional ma- 
terial, this type of information is not specific enough. It is not very 
info? ativc to know that a certain person has passed through a s.rdy 
material A and done somewhat better than the average student. It would 
often be much more interesting to know what he is able to do. 

If this is what we want, the two ether criterion -related as;. -Vicn , 
come into focus. The difference between them jlS some- 
times not vary large, and the distinction is often overlooked. If the 
particular course has started from and closely kept to a given goal de- 
scription, then the difference should usually not be of importance, (if 
the main goal is not quite outside th*. scope of the subject-matter; there 
may be courses, where the subject-matter is more or less irrelevant, 
but where the goal is to teach the students a particular study technique, 
for example; this may not explicity be seen from the particular course 
material used). The reason why it is important to distinguish between 
type a and type ^ is, hov^’cvcr, primarily that many courses lack a 
clear relation between a specified goal description and a closely related 
course content. It may be easy and tempting to choose a textbook of a 
course and”make a test on it” without pondering upon the relation of the 
book to a given goal description or a goal conception behind it. Then, 
however, we will lose our possibility of detecting defects in the goal- 
Q relatedness of the course. 
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In general then, goal -oriented assessments are in many cases to be 
'•^referred to the other two types: they arc definitively more informative 
than the norm -related ones, and they are often more relevant than the 
process-oriented ones. When wc deal with programmed instruction, it 
is quite obvious that we should usually prefer to work with goal -oriented 
as sessrhents , very closely connected with the terminal analyses started 
from. This statement may seem rather self-evident, but it has certain 
consequences that may not be so obvious and even seem a little unusual 
to a person used to the routines of educational measurements. 

The fact is that some of the measurement routines that have been 
built up arc closely related to the aims of norm -related measurements, 
and that these are not equally meaningful in the construction of goal- 
relatcd instruments. For instance, when we deal with norm-related 
tests, it has often been natural to increase the discriminatory power 
of the items by screening away such test items that show an especially 
high or especially low frequency of correct solutions. If one has the ge- 
neral aim of obtainii.g a clear dispersion in one^s group, this is a natural 
procedure. In a goal -related test, hov/cver, one's main demand is that 
a test item should be clearly releva t to a specific sub -goal of the goal 
catalogue, and that the test item should differentiate between those who, 
in a terminal situation, arc able to reach this particular sub -goal and 
those who are not. Let us suppose that wc have a test item which ful- 
fils the demands just mentioned. Should such an item get a very high 
frequency of correct solutions, then this cannot be an arguemnt to re- 
move it: the high frequency shows only that the instruction has been 
effective in this respect. Should the item get a low frequency of correct 
solutions, on the other hand, then this fact cannot be a reason for 
screening it out either: this may rather be an important indicator that 
the instruction on this point has to be improved. The discriminatory 
power of the items in the particular product group is thus not an adequate 
basis for screening out items, when we deal with goal-oriented tests; 
it is more important that the items discriminate between groups about 
which one knows, from other sources of information, that they have/ 
do not have abilities which correspond to the goal description (cf* Glaser, 
1963 and Marton, 1969). 

In general, then, we do not choose test items with general predictive 
symptom value, but items which arc closely related to specific goal de- 
scriptions; and we do not necessarily weed out items which most students 
are able to solve after going through the course. 
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14. 3 MEDIAL VARIABLES: BEHAVIOR IN THE INSTRUCTIONAL 
SITUATION 

The behavior in the instructional situation is studied in detail during the 
explorative^ individual try-out phase and from certain points of view 
(for example, with respect to correct vs, incorrect response behavior 
on individual instructional units) also during the group try-outs and in 
the final field study. 

14, 3, i Medial variables of achievement character: Analysis of correct 
and incorrect responses during the work with the program 

Among the most common aspects of the examination of student behavior 
in working with programmed materials, is error analysis . This is very 
reasonable both from a common sense point of view and from the view- 
point of Skinnerian theory. The risk is that the programmer stresses 
the counting of errors more than the analysis of errors* 

Instructional micro-units (’’didules*’) that have caused many errors 
should not be discarded without further investigation (in some cases and 
for certain purposes a somewhat higher percentage than usual may be 
acceptable and educationally meaningful), but these error -causing units 
should always be carefully examined, so that the programmer can come 
to as realistic conclusions as possible about the reasons for the increased 
error frequency. Some of the most common reasons are, of course, that 
the response demand has been ambiguous, that the language has been 
poorly adapted to the particular student group, or that the background 
behavior, built up earlier in the program, has not been trained to a 
sufficiently high level of stability and precision to ensure a correct 
solution in the specific task under study. In all these cases a revision 
seems to be the appropriate next step for the programmer. 

Even though the study of errors is important, we should never forget 
that the error reaction alone doc not necessarily tell us very much 
about the effectiveness of the specific didule where the error occurred- 
One of the reasons is, of course, that the correct answers are usually 
given as feedback. We have no simple one-to-one correspondence bet- 
ween the dichotomy "correct versus incorrect response" on the one 
hand, and the dichotomy "understanding of the task versus lack of under- 
standing" on the other hand. 

During the first individual try-out phase, at least, it may therefore 
be advisable (Gordon, 1963) to have the students mark off the units that 
they do not quite understand (whether they have answered the questions 
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involved correctly or not). . A simple question mark in the margin of 
the text or on a special sheet of comments would usually be adequate. 
This gives us four categories of student reactions: (1) ’’error with re- 
maining doubt”, (Z) ’’error without remaining doubt”, (3) ”no error, 
but doubt”, and (4) ’’neither error nor doubt”. It is obvious that didulcs 
of the first type have to be revised. On the other hand, the second type 
may sometimes be educationally fruitful. (Once in a w*hilc the most 
effective instruction may rcrjult from the student^s being allowed to 
make an incorrect response and having it clearly classified as wrong; 
this may be the case when wc arc dealing with a ’’spontaneous error 
tendency”, caused by strong interference from ca.rlicr response 
patterns. ) The main advantage of this analysis probably lies in the fact 
that we can identify those units where the student have answered 
correctly, but where they nevertheless register lack of understanding 
(the third category above), hi these cases the programmer may, for 
instance, have used too strong prompting techniques, a fault that a 
simple error analysis unfortunately cannot demonstrate. Such units 
should usually be revised, too. 

This kind of analysis is useful not only for linear programs, but also 
for the error -treatment program of the scrambled book ty|:)c. The lack 
of correspondence between understanding and correct answers may be 
quite frequent in some of these programs, if no extra checks arc made. 
It is quite possible that the student chooses the correct answer imme- 
diately in a multiple -choice situation, without understanding the correct 
answer (the other alternatives may be too obviously wrong, for 
instance). There is also no guarantee that the final correct answer 
given by the student is connected with und.c rstanding. It may simple be 
the only alternative left after the student has passed through all the 
error branches (without having understood enough of the error treat- 
ment given). 

If wc try our program on one student at a time and simply extend the 
number of steps at each point where errors occur, there is a consider- 
able risk that we end up with a very large (and inefficient) program. 

On the other hand, if we feel content with overall error percentage 
(making no revisions at all if these percentages fall within certain pre- 
established limits)^ we do not fully utilize the information collected. 

The first approach places too much weight on the behavior of single 
individuals which may or may not be atypical (and is too much influenced 
by chance), whereas the second approach gives us too little information 




- 14. 10 .. 



about individual items. Instead, a systematic mapping of error distri- 
butions and error types would usually be preferable, and as o rule this 
mapping should also include subject reports on understanding. A natural 
starting point is then a simple survey table, including both errors and 
reports of lack of understanding. A square -ruled background grid with 
students as row headings and didule numbers as column headings would 
be a simple, but adequate design (cf. Figure 14. 3). This survey table 
should be supplemented with a frequency list of deviant responses. 

These aids are good starting-points when the programmer has to 
ansv/er questions of the following tyi^es: (1) Do the errors concentrate 
on certain specific didules ("column comparison")? (2) Is there any 
good educational reason for this, or is it simply an indication that these 
particular units or some immediately preceding units should be revised ? 

(3) Do the errors concentrate on specific individuals ("row comparison")? 

(4) Arc these errors of such a systematic character that some prepara- 
tory additional course (remedial branching after a filtering test) would 
probably reduce their number considerably? (5) Do the same error 
types occur repeatedly for some individuals? (?) Could this best be 
avoided by more complete information in the main progi'am or by utilizing 
special branching for remedial instruction? 

For a summary discussion on the relation between item difficulty and 
learning effect we may refer to Jacobs (1963). The main thesis of 
Jacobs is that many variables influence the degree of difficulty without 
influencing the learning effect and vice versa, and that it may therefore 
be misleading to concentrate the study onesidedly on item difficulty (in 
terms of error frequency) as the only evaluation criterion. Our examples 
above point in the same direction. 

14. 3. 2 Medial Variables of Experience Character: Analyses of 

Attitude Development and the Degree of Subjective Difficulty 

In addition to such evaluation variables that deal with the final learning 
result (the terminal behavior) and such that deal with correct vs. in- 
correct behavior during the study process, it is often also desirable to 
examine the developme nt of attitudes in the instruction situation (or, to 
use another word, the general "climate of work"). Successive observa- 
tions of the study behavior during the course as well as direct attitude 
questioning at certain points of time are natural methodological 
approaches. Among questions in which one would usually be interested 
in such studies are: How does the student experience the self -instructional 
work situation? Is this felt to be more or less stimulating than the 
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Figure 14. 3 Typical structure of a simple survey table 
('*Err or - and- doubt distribution") 

In this simple hypothetical example we find a clear error concen- 
tration on unit 3 on the one hand (for almost all subjects) and for 
Subject 2 on the other hand (for almost all units). We also find 
a relatively large number of question marks for unit 5 in spite 
of the fact that this unit has not attracted any errors at all. 

These kinds of information are obviously better starting-points 
for revision than overall error percentages for total programs. 
For more detailed discussion, cf. the text. 
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traditional teaching the student is used to? And if it should be ex- 
perienced as more stimulating: Is this only a temporary phenomenon, 
some kind of novelty effect, or is it more stable, observable during a 
longer period? What attitudes to the particular subject-matter do we 
find under the particular conditions of prograrnmied instruction? Can 
we find any influence on the students^ attitudes towards studying in 
gene ral ? 

In the debate on programmed instruction, particularly during the 
early years, many participants seemed to have clearcut and general 
answers to such questions. Unfortunately, however, they were often 
more emphatic and certain than unanimous. Probably such questions 
can only to a very limited extent be given general answers. The answer 
for a given situation will depend both on the particular study course 
under work (there arc stimulating and dull programs just as there are 
stimulating and dull textbooks) and on the other experiences of the par- 
ticular student group (the "comparison” criteria). It should be no 
hindrance to our study of attitudes, however, that the answers only sel- 
dom can be generalized. On the contrary: it will be so much more im- 
portant to evaluate the specific course material, as experienced by a 
particular target group, not only from the point of view of achievement 
but also from the point of view of attitude development. 

Side by side with the attitude variables proper, it may sometimes 
be of value to study other "experience variables", especially the "sub- 
jective difficulty level" of the itcir^is. When constructing tests one 
commonly define the difficulty of an item in terms of frequency of correct, 
solution, but difficulty defined in this v’’ay is obviously not meaningful 
in the individual case. Let us suppose that the calculating tasks 
(1) 2 X 21 = ; (2) 3 X 34 = ; (3) 11x13=; and (4) 145 x 112 =; for the 

four students A, B, C and D result In the following distribution of correct 
and incorrect solutions: 



We note that Task 1 is solved correctly by all. Hence, it has the p- 
value of 1. 00 and is then, using common ^est -construction terminology, 
an "easy" task. But is it equally easy for student D as for student A ? 

If we define difficulty level in terms of frequency of solution, the answer 



1 



2 3 4 



A C C C C 

B C C C E 

C C C E E 

D C E E E 



C = correct solution 
E = error 
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is ^yes^ - or rather: then the question is meaningless. But very pro- 
bably the degree of ’’subjective difficulty” is greater for student D than 
for student A. We might get some idea about whether or not we are 
correct in making this guess, by examining the amount of time for so- 
lution and/or by means of direct student ratings on difficulty, and it 
should be of interest to relate such variables to both achievement va- 
riables and common attitude variables. (Our discussion of this point 
follows ideas in Jacobs, 1963). 

14. 3. 3 Special Aids: Didactograms 

If the program, is non-linear (for instance, using parallel tracks, re- 
medial loops, by-pass arrangements, etc. ), the simple survey table 
is sometimes not the best possible starting-point for a study of error 
frequencies on various items. There are two main reasons for this: 

(i) Error frequencies should usually be considered in close connection 
with their position within the branching system, and (2) we also need 
information on frequency of use for the various tracks or branches in 
the system.. Hence, in these cases, it is often useful to draw special 
diag rams. 

The basic outline of such a diagram will be stable for a particular 
program version, irrespective of the student reactions, and will show 
the sequence of the didules, including all available branching possibi- 
lities, As starting-point for these diagrams it is often suitable to use 
the ’’didactoplan” constructed earlier during the preliminary decisions 
on sequence fcf. above). On this background diagram, we may fill in in- 
formation about the individual's actual working route as well as his 
working results on the single units (correct and incorrect responses 
symbolized, for instance, by means of plus and minus signs). Such a 
program outline, filled in with student behavior registrations, might be 
called a ’’didactogram”, and it gives us a relatively survcyable map of 
the interaction between the instructional system of the program and an 
individual (cf Figure 14.4). 

Data from, several individual diagrams can be brought together in a 
’’ group diagram ”, on which information about the percentage of correct 
solutions for each single response demand, as well as information about 
the number of individuals using various possible tracks within the branch- 
ing system, can be registered (cf. Figure 14. 5). 

Such a diagram can be used by the programmer for many different 
kinds of decisions. To take only one example; if may often be shown 

62 



that Gome of the branching possibilities have not been used at all. If so, 
it is natural tack of reviGion to get rid of these tracks or give them 
some other form. Such an analysis is important, for instance, when we 
work with multiple -choice arrangements and subsequent error instruc- 
tion connected with the various possible error choices. Perhaps some 
of the error choices never attract the attention of the students, and the 
special error instruction included within the program is then dead wood. 
Unfortunately there is seldom any evidence that the productive authors 
of scrambled books have exanoined their often very bulky error instruc- 
tion in this way. 
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Figure 14. 4 A diagram of branching use and response correctness 
for a single student 

("individual didactogram"): An example 
Background symbols ; 

(The background structure is the same in all diagrams representing 
the same course version) 

F = filter function 

T = terminal testing function 

O = ^ didtile that all students should go through 

^ = a didxile in a special, additional course with remedial function 

^ = a didule in a special, additional course containing optional 

"enrichment" 

O = a didtile in a parallel branching system 

(Q a didule with connected error treatment (two error alternatives) 
Working symbols (individual data): 

the route followed 

+ = correct solution 

— = incorrect solution 

The diagram is read in the following way: 

The subject gives a wrong answer to the introductory filter unit 
(no. 1) and is then directed to go through the special remedial items 
1-4, the response demands of which he answers correctly. Since he 
also gives a correct response to unit 5, he does not need any further 
remedial treatment, but is allowed to proceed directly to no. 8. 

Among the possible parallel tracks (covering units 10..12) he chooses 
the one called Alternative B. He makes one error choice in each of 
the multiple -choice tasks (15 and 16). He chooses to go through the 
optional enrichment sequence (17-19). After that he solves the ter- 
minal test item correctly. 
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Figure 14. 5 A group diagram branching use and response 
correctness (''group didactogram") : An example 

Background symbols : See Figure 14. 4 

Working symbols: Figures without parenthesis give the percentage 
of correct responses ("freouency of correctness''). Figures with 
parenthesis give the percentage of students who have used a speci- 
fic passage ("frequency of use"). 

The diagram is read in the following way : 50 per cent, of the students 
answer unit 1 correctly, and these students are directed to go for- 
ward directly to number 5. The rest of the students go through the 
remedial items 2-4 (in which thG resulting correctness percentages 
are respectively 85, 90 and 95 per cent. ) On item 5, correct an- 
swers are given by 60 per cent of tlie students; these students are 
allowed to go directly to number 8. Tlie remaining 40 per cent, pro- 
ceed through numbers 6 and 7, etc. 

Comment s: The correctness percentage is fairly high. for most of the 
units. An exception is number 11 in Alternative B; this unit should 
therefore be given special attention. The units with filter function 
(i and 5) have a moderate correctness percentage, which usually 
can be considered to be adequate (if all students should answer a 
filter unit correctly, the appended remedial sequ nee would be un- 
necessary). The error-treatment units 15 and 16 give uneven values 
(D 15 has few correct answers, and D 16 has uneven distribution 
between error alternatives) and should therefore be especially 
examined. Finally, it is, of course, a good sign that the terminal 
unit with testing function (20) has a high correctness percentage. - 
Frequency of use : The optional choice of parallel sequences gives 
an uneven distribution (almost all students choose Alternative B). 
Here revision is probably indicated. The distribution in the choice 
of enrichment courses, however, does not seem to give any reason 
for revision. 
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1 4. 4 ANALYZING RELATIONSHIPS BETV/EEN STUDENT BEHAVIOR 
V/ITHIN AND OUTSIDE THE PROGRAM 

The final criterion of the effectiveness of a program is, of course, the 
Gtudent^s terminal behavior. As was stated above: If this reaches accept- 
able levels within reasonable time periods, we need not be so concerned 
about the details of the behavior within the program. However, when 
the terminal behavior is not fully adequate or when the time used seems 
too long, an analysis of the student^s behavior v/ithin the program is 
needed to show uc the place where revision is most indicated. About ■ 
these general ideas there seems now to be a fair degree of agreement 
among programnaer s . However, two deficiencies stand out in many re- 
ports on program development. Gometimes the programmers are so 
exclusively interested in the error percentages within the program that 
all other considerations, even those concerning the terminal behavior, 
seem to be forgotten. In other cases attention is paid both to final tests 
and to the error frequencies within the program, but the programmer 
does not care to study any particular relationships between student be- 
haviors within and outside the program, even when these relationships 
should give him important additional infor: lation (such as, for instance, 
would often be the case in branching programs). In the following we 
will give a few brief illustrations of relationships between student be- 
haviors within the program and outside the program that should be of 
interest to the programmer. 

14. 4. 1 ^formation Density and Information Speed of a Program 

As has been pointed out (Bjorkman, 1963) the information value of a 
unit of information can be defined as the difference between the pro- 
bability of correct answer before and after the communication. Since 
the single units (didules) of a program often do not contain complete 
terminal reactions, however, but have other purposes as steps on the 
way towards, such terrxainal reactions, it is often unrewarding to study 
the information value (in this sense) of the single units. (This value 
may be mairimal in the stated operational m.eaning without any gurantee 
that we have come closer o any of the terminal reactions. ) On the 
other hand, it should be quite meaningful to study the special tasks, 
sometimes called ’’criterion problems'*, which test the student^s ter- 
minal behavior after he has gone tlirough an instructional sequence 
of related units. Even when formally built into the program proper, 
these criterion problems are equivalent to tost items and may bo 
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considered to represent ‘‘behavior outside the program” in th<i sense 
intended here. If we examine the ability of the students to solve these 
criterion problems before going through the instructional sequence 
(error percentage A) as well as their ability to solve them in their na- 
tural contc::t, that is, after going through the sequence (error percentage 
D), v/c shall obtain more meaningful measarements of "uncertainty 
reduction” (error percentage A - error percentage B). 

Let us assume that, for each new concepts we introduce we have 
at least one instructional sequence, and at least one criterion problem. 
The uncertainty reduction of the cri. xion problem divided by the num- 
ber of units (didules) included in the sequence then gives us a measure 
of the “information density” of that particular sequence. The infor- 
mation density ox the total program may then be defined, in principle, 
as an average of the density of the separate micro sequences: 









A.-B, 



A -B 
n n 



D, 



D. 




n 

= the error percentage of the student group under e::amination on 
criterion problem i before the relevant instructional material has 
been studied. 

= the error percentage of the student group under examination on 
criterion problem i after going thorugh the instructional material. 

= number of units (didules) in the micro sequence that leads up to 
criterion problem i (and the purpose of which is to tcacli the stu- 
dent the concept C^). 

= bo.sic concept (represented in the program by the i^^^ criterion 
problem and the i^^^ instructional sequence), 
n = number of basic concepts included in the program. 

‘ Notc > Obviously a formula of this type only gives one aspect of the 
complex plicnom.cnon of information density as intuitively conceptualized. 
For instance, it does not consider the fact that various conceptual units 
can make widely differing demands on the "intermediary processes”. If 
these demands could be expressed quantitatively (and average jtxdgmcnts 
from a group of raters might perhaps be used in this connection - at 
least for a first appr jximation), a weighting of each quotient (A^-B^/D^ 
with respect to the demands on the intermediary processes might 
"improve” the formula, that is, mean that one more possible criterion 
of step size has been taken into consideration. 
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If wo exchange the information about the number of unit;3 (diduics) 
in the micro sequences D., , D.y . . . against the time the micro 

sequences take (T^ T^, T , ... T^^), v/e would obtahn a measure of 
•’information speed” inctc.ad of a measure of “information density”. 

Since one of the most basic criteria of educational cCfcctivcnc s s is liow 
much information the student absorb per unit of time (not how much 
information liac been poured ouc from the mouth of the teacher or 
otherwise been potentially available for absorption!), such a formula 
would give us one starting- point in our assessment of the educational 
effectiveness of the program*, [n any ease it ought to bo of interest to 
be able to compare this ■mo<?.suro with other data about a program. It 
is hoped that comparisons between different operational program 
characteristics will gradually give us a clearer picture of what is 
essential and what is los.s essential in program construction. 



14. 4. 2 Choicc-.Fbint Relationships (Gurvey Tables of the Distribution 
of Achievement Patterns in Filter Units ) 

The measure of "information speed” ju5:t mentioned combines informa- 
tion about time used (a "medial” variable) with information about pro- 
babilities of correct responses before and after the study (“initial” and 
"terminal” variables). Another combined evaluation criterion will be 
of interest when examining br.anching programs where the students, on 
the basis of their achievement at a choice- point (in a filter unit), arc 
led different ways througli t]\o program. Simple error frequency data 
for these filter units arc, of course, insufficient as a basis for the re- 
vision, since we arc not here concerned merely with minimizing the 
number of errors. Instead, in these eases the aim is to lead the student, 
by means of the diagnostic indications given by the filter unit, to the 
particular study material that is most approoriatc to him. This means 
that the student'^G achievement on initial test (pre-test), filter unit, 
and terminal test have to be e:aarnined together. It is obvious that cer- 
tain combinations ("achieveraont patterns”) arc acceptable, whereas 
others Siiov/ faults in the progr.am and, hence, indicate v/here revisions 
arc needed. (A demonstration of this was given by Marklo, 1963. Some 
of Marklc's idca.s will be incorporated in the following illustration. ) 

As an aid in this c.xarnination the progra.mmor can utilize a survey 
table of the general kind demonstrated in Figure 14. 6, The eight possible 
^•ncliicvcmcnt patterns” are placed as row headings, and the filter units 
pR|(^thc program as column headings. The distribution of the students 
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over the various achievement patterns is then built up within the table 
by entering tally marks. 

L.et us take a brief lool: at some typical achievement patterns when 
the program contains filter units of the general type that sort the stu- 
dents into tv/o sub-catcgorics at each choico-point , one of which passes 
through only part of the total ;iumber of units included in the. total pro- 
grams. This formulatioii is a very general one, describing a formal 
branching structure, and it therefore covers at least two main situations 
that the educator v/ill often consider to be quite different. In the first 
of these, the main group of students goes through all unite, v/hilc a 
smaller group of students is found at the choice -point to know already 
some of the material that wall bo treated and is therefore directed to 

sldp some part of the program ('’shortening of learning sequence for 

< 

liigh-achicvcmcnt group"). In the other situation, the main group of 
students goes through the shorter sequence, while a smaller group of 
students is found at the choice -points not to know enough about materials 
covered in some carlici' program and is therefore directed to go 
through an additional "remedial loop" ("lengthening of learning sequence 
tor low -achievement group"). The pattern symbols used in the following 
are explained in Figure 14. 6. Only some of the patterns included in 
the figure will be commented upon here, and, for the sake of brevity, 
only faults of the program proper arc rrentioned (th t is, we assume 
that pre- and post-tests arc reliable and valid); 

A. I FD T 
r r r 

In th’s case the student answers correctly (r) both in the filter unit 
(FD) within the program and in the corresponding parts of initial test 
(I) and terminal test (T). Students with this pattern will skip the special 
part of the program immediately following the particular filter unit. 

In other words, they go on the quicker track (whether this is considered 
to be the main track or a track for selected minority of high-achievers). 
No criticism can, of course, be directed against the fact that this 
pattern appears^ One indication for revision may come up, however. 

If the number of students with this pattern approach 100 per cent in a 
representative sample of the target population, the additional part of 
the program in superfluous. 

C. I FD T 
r V/ r 

If the tests arc assumed to be valid (an assumption we made above 
for brevity of discussion), this particular pattern indicates tliat we have 
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Achic vonH'Jil 
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FD, 




i-'O , 




FI t 


A. 1 FD T 
r r 1 - 












B. J FD T 
r r w 












C. I kd T 
r w r 












D. 1 FD T 
r w 












E. 1 FD T 
w r r 












F. r FD T 

V/ r w 












G. I FD T 

w \v r 












H. 1 FD T , 

W W \V 













Rt*adi3i>^ kuy : 

I = initial teat 
FD = filler unit (filter clidulo) 
T * ^ toriTiinal tost 
r = ri^dit 
w = wronji 



Figure 1 4, 6 Possible table arrangemenc in analyzing the 
distribution of achievement patterns in filter 
units (*‘choice-poiiit relations table") 
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had an inadequate filter unit which directed the student through an 
extra part of study material that he did not need. The response demands 
in the filter unit may have been formulated in such a way as to lead the 
student astray, or the filter unit may simply be too difficult for its 
purpose. Hence, in this case the revision should focus on the filter 
unit. 



F. I FD T 



w r w 

This combination can be an important indicator of insensibility on 
the part of the filter unit. The student who does not know the pertinent 
facts from the beginning is not correctly sorted by the filter unit and, 
hence, does not get the additional study he needs. This may happen 
quite often, for instance, if the filter unit uses multiple-choice tech- 
niques with less adequate alternatives (or if, in the filter unit, the 
student is tested on discrimination behavior only, whereas he is tested 
on construction behavior in the terminal tests). As was the case in 
pattern C, the revision should be directed at the filter unit, but in 
pattern F the revision has the opposite purpose of increasing the diffi- 
culty of the filter problem. 



G. I FD T 
w w r 

This combination is usually a good sign. The student shows error 
in the initial test and in the filter unit, but he can solve the terminal 
problem after the additional training supplied by the extra part of the 
program. One possibility of revision should, however, be kept in mind. 
The programmer should ask himself whether or not the ’’additional ” 
program part ought to be incorporated within the main part of the pro- 
gram. If the number of students with G patterns approaches lOO per 
cent- in a representative sample, such an incorporation is clearly 
indicated. 

The tabular and diagrammatic arrangements demonstrated in the 
present section should be considered only as examples. Many other 
types are possible and suitable to aid the programmer in his attempts 

to deal systematically with the data from the try-out phase, and each 
programmer should, of course, design his aiding devices to fit his par- 
ticular needs in each specific situation. Our main purposes have been 
(a) to emphasize the relative complexity of the evaluation task and the 
risk of using one-sided criteria in the try-out process, and (b) to show 
some of the possible ways in which the programmer can equip himself 



71 



some 



14 . 23 « 



with simple aids in order to make his evaluations more systematic, 
complete, and efficient. It is felt that a more widespread use of such 
systematic and comprehensive evaluation schemes would also consider- 
ably enhance the accumulation of basic knowledge in the programmed 
learning field. 
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15. EXAMPLES OF PROCEDURES IN REVISION OF PROGRAMS 



We have already discussed a series oi steps in revision, arising from 
different test results, when we were dealing with the various evaluation 
criteria above* In the present section we shall briefly review different 
combinations of criteria which to a greater or lesser degree call for 
revision and also mention a few special measures which can occasion- 
ally be indicated. 

15. 1 SURVEY OF PRELIMINARY INDICATIONS FOR REVISION FROM 
CERTAIN COMBINATIONS OF CRITERIA 

Figure 15. 1 gives a simplified survey of the extent to which different 
combinations of criteria indicate revision. Such a survey must obvious- 
ly be fairly scherriatic. In practice, it is not always easy to determine 
what are ’’high'* or "low" error rates or decide, for example, how a 
"limited revision’’ should be designed. But as a theoretical outline such 
a schematic survey can perhaps nevertheless give a lead and counter- 
balance the one-sided consideration of a single criterion of which some 
programming theoreticians arc guilty. 

Let us comment briefly on some of the criteria combinations. (For 
abbreviations used, see the explanations in Figure 15. 1. ) 

All combinations with a high terminal error rate (TE^^) obviously 
definitely indicate revision In these cases the goal set has not been 
attained. The program is, to use very obvious terminology, simply not 
"finished". If, in addition, we consider the student^ s attitudes, we get 
yet another indicator, and the difference made in the survey between 
"maximal" and "strong" indication for revision is directly dependent 
on the difference between programs as experienced by students with 
negative and positive attitudes respectively. 

A program which not only misses its performance goal but also 
created an unpleasant study situation does not justify its existence. In 
this case it makes no difference whether the error rate is high (as in 
the first criteria combination) or low (as in the second). Programmers 
who discuss revision only on the basis of the students^ errors during 
the course of the program risk accepting a prbgram of the type 
MEjA^TEj^, which is here shown as indicating a maximal need of revision. 

Everyone would agree that criteria combination 8 (ME. A TE.)is 

Ip i 

advantageous, and that a program with these characteristics scarcely 
Q needs revising. (For the sake of simplicity, we are disregarding certain 
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other variables, c. g, teaching time, which is in many cases important, 
but which is less so compared to the final performance and working 
climate. ) We have also accepted criteria combination 7 (ME^^A TE^). 

The error rate during the working of the program is admittedly high, 
but this has resulted neither in bad performance nor a negative attitude. 
The difficulties can have resulted from a low degree of support stimula- 
tion or a teaching method which failed to explain errors satisfactorily, 
but these difficulties are not such as to have an overwhelmingly de- 
terrent effect. It is possible that this combination is comparatively 
rare, but when it does occur, it should hardly be rejected, despite the 

ME, factor, 
h 

Finally, combinationo 5 and 6 are characterized by a negative attitude 
together with a good final result. In one case, (ME^A^TE^), the negative 
attitude is probably connected with the error-reaction frequency. Here 
it is possible that a limited revision to reduce the erro: rate would aldo 
produce a better attitude climate. In the other case, it s more probable 
that the negative attitude is related to the fact that the program is 
experienced as monotonous and uninspiring. In this situation, an attempt 
should be made to increase motivation and reduce monotony (cf. our 
discussion on motivation above). 

In a particular situation, the occurrence of different combinations 
depends not only on the character of the program, but also on the ge- 
neral working climate of tl:c classroom, or school and on the student as 
an individual. Some programs probably give rise to widely different 
criteria combinations for different individuals which may be of interest, 
for example, from the point of view of personality research and social 
psychology. From the point of view of educational technology, however, 
the general aim must be to construct the program or system in such a 
way that as many of the target students as possible come up with com- 
binations 7 or 8. 




74 



- 15. 3 - 



Figure 15.1 Outline of principle indications for revision in certain 
criteria combinations 



i Criteria 
1 combination: 

j 


Preliminary indication; i 


1 :L ME. A TE, 
h n h 


. 

Maximal revision indication 1 

1 ‘ 

1 . - - -! 


Z. ME^A^TE^ 


1 

) 

j Maximal revision indication 1 

1 

_ . - -1 


3. ME. A TE, 
h p h 


1 

Revision strongly indicated j 

_ . . J 


4. ME, A TE, 

1 p h 1 


Revision strongly indicated 

1 . 


— — 1 

5. ME, A TE, i 

h n 1 j 

. .. J 


1 

! Limited revision indicated 


6, ME,A TE, ! 

In 1 


1 — — — ^ ■ ^ 

1 

Limited rovdsion indicated 


7. ME, A TE, 
h p 1 


Revision not indicated 


j 

j 8, ME,A TE, 

! 1 P 1 1 


Revision not indicated 

i 



Symbols: 

ME = medial error rate (% of errors during the instructional program) 
TE = terminal error rate (% of errors in final test) 

A = attitude 

h, 1 = high, low 

p, n i: positive, negative 
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15. 2 SPECIAL PROCEDURES 

15. 2. 1 The Question of the Student^s Capacity for Independent Work 

and the Possible Alteration of the Boundary between Educand- > 
System and E due ator»>Sys tern 

As is shown in Figure 15, 2, the change from one type of teaching situa- 
tion to another often involves a shift of the boundary between the edu- 
cator system and the educand system. An extended educator system may 
be valuable for the rapid learning of a certain subject-matter. On the 
other hand, when it is a question of developing the capacity to cope in- 
dependently with subject-matter which is hard to learn, a comprehensive 
educator system would often be a hindrance. If the try-out shows that 
the learning of certain subject-matter takes more time than is desirable, 
one should consider changing over from a less detailed learning process 
to a more rigidly controlled one. If the aim is to induce the ability to 
treat subject-matter independently, and if the try-out seems to indicate 
that this goal is not being reached, a shift of the boundary between the 
educator system and the educand system in the opposite direction would 
be indicated. 

15. 2. 2 The Question of How the Student Accustomis Himself to Active 
Cognitive Handling of 3ubj ect -Matter and Possible Limitation 
of the Number of External Response Reactions 

External response reactions are considered valuable as a guarantee that 
the student actively handles the subject-matter presented to him. Is 
there a possible risk that the student accustoms himself to the special 
working technique in such a way that he avoids active cognitive working 
when there is no explicit response demand? Can the technique some- 
times involve an unnecessary loss of time (for those with already 
established habits of study)? 

If the try-out indicates a positive answer to one or both of these 
questions, one should possibly try a technique which is a compromise 
between the program's continual demand for external response reactions 
and the ordinary text-book^ s lack of such a demand. The program can, 
for example, be so constructed tliat a response which generally need 
only be mental ( implicit) is demanded of the student. From time to 
time (at irregular intervals), however, a demand for an external state- 
ment occurs. Cook (1962 a), who suggested this type of technique, 
thinks that by this means the individual might be forced to keep his 
covert responses alert and so save time. 
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l^Mgur e 1 5 , Z Outline sketch of tho varying boundary between educator 
system and cducand syst(*m 
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A schematic diagram of the reaction between student and program 
when using such a technique (with partial demands for explicit statement 
of responses) is given in Figure 15. 3. 



15. ? 3 Special Measures against '^Bo r edom *' 



We discussed above some general steps which could be taken to reduce 
monotony and increase motivation. Should investigation show that the 
students find the program monotonous and boring, the revision should 
aim at an oven greater use of these tactics to give variety. In this sec- 
tion we shall give a few brief hints on other possibilities. 




(1) Adaptation to the stude nts'" pr evious knowledge has perhaps not 
been worked out with sufficient care. As illustrated in Figure 15.4, the 
student'^s attainment of the skills relevant to a certain goal is seen as 
a continous process, which has all jady gone on, to some extent, before 
he is faced with the particular study material. If he is aware that ho is 
being taught about things he a.lready knows, he will very often experience 
boredom. In this case a more thorough student analysis should form the 
background for a revision of the program. 

It may become apparent that the previous knowledge of the target 
population has been seriously underestimated, and then the program 
should be shortened - either by cuttiiig out certain sections in the more 
elementary parts of the program or by "increasing the size of the steps" 
(reducing the number of application- and prac tic e -tasks). But it can 
also happen that the underestimation applies only to part of the target 
population and in that case consideration should be given to changing 
from a linear to a branching program. 

The most effective teaching should, in theory, be that which starts 
out from an individual analysis of the initial repertoire, i. e. which 
begins by testing and analyzing the individuals previous knowledge and 
tendency to errors and then teaching only those points which arc shown 
to be lacking - with emphasis on individual difficulties. The fact that 
the programmer for practical reasons finds it difficult to follow such a 
procedure completely does not prevent him approaching this technique, 
since the number of individual variations important for teaching is rela- 
tively limited in most groups of students. (One works then with a number 
of "streams" in which the students are placed after an initial filter test. ) 
It may be thought that student analysis is troublesome and timecon- 
suming, and it is often neglected in preparing the contents of the curri- 
culum. However, the amount of work that one imagines one saves by 
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of the answer {ct , turns the machine forward one frame the correct answer (K^). In 

Didule 2, however, when he turns the machine forward (^02a^’ instead the instruction to write 

down the answer (l^). Only after he has done this (^r^) is he given, after a further manipulation of the 
machine, the correct answer (K^). 



- 15, 8 ~ 



Figure 15. 4 Theoretical illustration of relation between initial repertoire 
and contribution of the program 
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omitting this procedure will be lost over and over again, partly in 
lowered motivation among the students, with the consequent lower 
effort they put into their work, partly in an unnecessary expenditure of 
time and study-material when one teaches what the students already know. 

(2) Alteration of tempo of presentation. As far as the tempo of pre^ 
gentation is concerned there are three main possibilities: student- 
controlled speed, machine -controlled speed and the combination student- 
and -machine -controlled speed. Student -controlled speed is the most 
usual, and it has the obvious advantage that each student has as much 
time as he needs to think. However, one can imagine (especially when 

it is not a question of concept -learning or problem-solving activity, 
but of a motor skill) that the student-controlled tempo involves a risk 
that the student will stay too long at a low level of skill. This is likely 
to lead to boredom, since clearly increasing demands are not con- 
tinually being made on the student. Should a try-out show this to be the 
case, a situation should be considered in which the student is spurred 
on to greater efforts by a machine-controlled increase in speed, or 
which uses a student -machine system where the average speed of the 
machine is regulated by the student^s skill (cf, SAKI). 

(3) Variation of didule components. We normally work with a fixed 

series of didule components and student responses: information or 
'*stimule" (3), response-demand or "respule" (R), student response or 
’'response*' (r), feed-back of correct answer or "corrule" (K), This in 
itself, as we pointed out above, gives a good dynamic variety (S-R-r-K, 
S-R-r-K etc, ), with greater variation than the usual text^s monotonous: 
S-S-S-S-S etc. However, if we work only with series of the type S-R-r-K, 
even thiu can naturally gradually become monotonous and boring. There 
is a particularly great risk that this will happen when, in addition, there 
is a great number of consecutive didules which contain very similar S, 
which is often necessary in the training of skills (e. g. in languages), 
where the student, by persistent practice, has to get as close as possible 
to the ideal performance (e. g. perfect pronunciation). If investigation 
shows that boredom is experienced by the students, one should consider 
interrupting the working sequence at suitable places with differently 
arranged component sequences. If, for example, in training of skills, we 
occasionally intensify the r-K c ombination by "extended" sequences, this 
can serve a useful purpose both for effective work (the ideal performance 
is attained more easily perhaps when the whole task need not be repeated) 
and for the students^ need of "something different". This type of sequence 
- e. g. S-R-r-K-r-K-r-K- is of course only one of many possible » 

variations. 81 
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16. COMPILING THE PROGRAM MANUAL 

Program consumers need c ertain information from the program pro- 
ducer to help them choose their study material. As we have already 
pointed out, this information should be readily available in a program 
manua l. The writing of this manual should be the final task of the pro- 
gram constructor. 

It is naturally not a great deal of use to describe in detail what such 
a manual should contain. The most suitable oice will vary greatly 
from program to program. But certain catf ^ ies of information are 
usually necessary in one form or another; 

(A) Description of goal and students 

The aim of the program and the use planned for it should be clearly 
stated. An attempt to attract customers among a wide and varying 
public by using vague and all-embracing headings is easily seen through 
and is not a good advertisement for the vendor of the program. The 
previous Icnowledge needed is stated an preferably specified by means 
of a pretest. Similarly, the purpose of the course is clearly defined by 
the final test. (For pre- and post-tests, see E below. ) 

(B) Character of development v/ork 

(a) Those responsible for the programming and testing should be named. 

(b) The goal analysis on wi ich the program is based should be presented. 

(c) A summary of the other stages of the preliminary v/ork can give 
valuable information. (To what extent has a thorough student ana- 
lysis been carried out? Etc.) 

(d) ^ revision and try-out processes should be given in outline. ’ 

(C) The structural characteristics of the program 

should be briefly described and illustrated with extracts from the pro- 
gram. It is possible that in time a standard set of descriptive categories 
may be used as a basis for such a description. Fry (1963, p. 199 ff» ) 
exemplifies a scries of possible bases for classification. Some of those 
it would be of interest to give are: 

(a) sequence characteristics , including main flow structure, 

(b) general didule characteristics , and 

(c) special characteristics of the components , i, e. of stimules, 
respules and corrules. 

(D) Characteristics of the subject-matter 

It should be stated here to what degree the program concurs with the 
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curricula concerned, and also how far and in what way it differs from 
alrca.dy existing tc::tbooks (or programs) in content. 

(E) Pro gram effects 

This is, in a way, the most im^vortant category of information. It is 
here that the intending purchaser of the program is informed of the 
demonstrated teaching effectivity of the program and of the conditions 
under which this empirical demonstration was carried out. In other 
words, the manual should contain the following information: 

(a) Scope and conditions ..f the field try>out. This should be stated 
in some detail and comprise not only information about the research 
sample (its size, age of students, intelligence, previous knowledge etc.), 
but also external circurnstancc^s both iis regards the teaching situation 

(c. g. , how time was allotted, possible teacher c o- operation) and as 
regards the test situations, 

(b) The preliminary and final test s used in the field try-out should 
be described in the manual an^l in additi »n the complete tests should 
bo available from the c.mslructor.s or the publishers if desired. (Tests 
arc sometimes sold together with sclf-instructi onal material. Accord- 
ing to * 'Prog rams '6 3 there wore then final tests for 46% of commer- 
cially available programs, but preliminary tests for only 14%. Were 
the ioublishcrs afraid tliat the tests might show that the student did not 
need the program in questi n?) 

(c) The final results of the field try-out should obviously be clearly 
stated. It would be an advantage to give this both verbally and in the 
form of adequate statistics. It should be possible to compare the stu- 
dents^ final performance with their previous knowledge, and there 
should be supplementary information about the amount of study time 
used. If other aims than simply the learning of subject-matter have 
been mentioned in the description of the goal of the course, the manual 
should also give an account of results of the try-out which confirm that 
these aims have been attained. Otherwise, the manual should clearly 
state that these points in the general goal description have not yet been 
confirmed by empirical results, and that therefore the effect of the pro- 
gram as far as they are concerned is, for the time being, hypothetical. 

The task of producing a program manual is the responsibility of the 
program producer and publisher. This can hardly be considered an 
excessive burden, provided that the program constructor has really 
developed his program empirically and carried out a proper field-test. 
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Unfortunately, this kind of account is at present supplied with very few 
of the commercially available programs. This is not just due to an un- 
willingness to do the extra work involved in preparing the manual. It 
is mainly because many of these programs have obviously not had a 
thorough try-out in the field, a fact which may well be obscured by a 
flood of sales-talk about continuing research and delighted students, 
but which would look bad in print. When, in one enquiry, publishers of 
programs in the USA were asked about the size of the groups on which 
the programs had been tried out, over 80% avoided answering precisely 
that question! (Cf. ’’Programs ^63”, p. xiii. ) In an examination of 
about 700 programs in 1967 at Northeastern University, it was found 
that only about 30% were accompanied by any information about field- 
tests. (Cf. Markle, 1968, p. 17.) 

So it is important that program consumers are aware of the demands 
they should make, and that they do not hesitate to make them. Reviews 
such as ’’Programs ^62 " and "Programs ^63" are undoubtedly useful in 
this respect, in that they give - without further evaluation, it is true - 
the nature and scope of the try-out. Publishers who are for^^ed to leave 
a blank as far as this is conerned can scarcely avoid wondering what 
it will mean for their sales if their competitors can give fuller informa- 
tion. In addition to such survey lists, a standard forum for regular 
reviewing would be desirable. To take an example from a somewhat 
different field, one can point to the influence which Buros^ "Mental 
Measurements Yearbooks’', with their i^ersistent pointing out of lack 
of reliability and validity checks , have had. 

But apart from this, every buyer of a program naturally has the 
right to demand detailed information from the seller. Advertising ge- 
neralisations are not much use to the consumer. The Bulletin of the 
National Association of Secondary School Principals (Belton, 1962) 
gives examples of a number of questions which a head teacher should 
put to a program salesman who has no program manual to show. Their 
tone and general tendency can be seen from the following (considerably 
shortened and modified) version; 

"We thank you for your offer to demonstrate at our school the pro- 
grammed study courses which your company is at present marketing. 
Our study material committee will be glad to see you next Monday at 
3;30 p. m. Our time, however, is limited, and we should appreciate it 
if, when you visit us, you would be prepared to answer briefly the 
following questions about each of the programmed courses you intend 
to demonstrate; 

1. Is there a teacher'^s manual or guidebook with instructions for using 



the course? 
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2. Arc the goals stated in detail in a description of the course? 

3. Has the course material been produced as a result of a series of 
try-outs and revisions? If this is the case, where can we find a 
description of this devciopmci:t work? 

4. Who were those responsible for the development and trying out of 
the course? What are their qualifications? 

5. To what extent has the completed course been tried out in the field, 
and where can we find an account of how the field trial was arranged 
and the results of it? 

6. Which schools or organizations can we ask about their experiences 
in using the course? 

7. Is the course accompanied by a pre-test and a post -test, or can 
such pre- and post-tests be ordered separately from your company? 
Where can we find out about the construction and evaluation of 
these tests ? 

8. What is the approximate average time taken for a student to complete 
the whole course? 

9. Is the study material avialable both in book form and for machine 
presentation? Is the study material consumable, or can it be used 
again? 

10. What is the cost of the material per student hour? Does your com- 
pany allow a discount for a first purchase to try out the material? 

When we have this information, we should be in a better position to 

decide on purchasing. ” 

If, after receiving such a letter, programs salesmen with deficient 

programs fail to turn up for the propsed demonstration, no harm will 

have been done. 
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17. assp:ssment and comparison of existing programs 




Wc have previously studied .self - in struct! on a J material mainly from the 
point of view of the constructor and how the tcachiiig program is success- 
ively re-examined, tried out and revised. Finally, wc shall treat the 
program in its finishcM..! state, above all with the foilow'ing question in 
mind: How can we decide whether a toacliing program fulfil.s reasonable 
demands and fits into a certain tea, chine situation? 

This is mainly the ;:-»rogra,ni eonsumcr^s conccjrn, but the question is 
also one for the pr-v^gram constructor. It usually turns up when the pro- 
gramming group has reached the poii^t when analysis of pre-requisites 
has been made, i. e. when one is clear what the goal is and \vha.t ty|:>c of 
students one will be dealing with. The questions one should ask oneself 
at this stage (but scarcely before arc: Is there already a program on 
the market which corresponds to the goal which has been specified and 
which is aimed at the same group of stu-.lcnts? Is there nevertheless a. 
sufficient reason for a new progran“i? 

(Figure 17. i shows schematically some factors in the situation about 
which the programming group ha s 7-nake a. decision. ) 

In the evaluation of a program, two kinds of judgment have to be 
made: (1) judgments based on the program's ’‘pheno-charactcristics ** 

which can be observed by a visual examination and (Z) judgments based 
on the program's ’'functional cha.racteristic s “ , i. e. on experience of 
how the program works in practice. 

Questions about ”phcno-criteria’‘ in their turn are of two types; 

(a) Has the content of the program just those qualities v/hich make it 
fit for the teaching purpose in question? (b) Does the structure of the 
program conform, with the ’'theory" of 2 :^rograramcd teaching (or with 
our knowledge of the principles of human learning)? 

Obviously, in judging a progra.m, one can easily come to certain 
conclusions by looking through it oneself, A new program should always 
be examined as throughly as a now textbook and primarily with regard 
to its c ont cut s and scope. 

On the other hand (as Rothkopf, 1962, among others, has emphasized), 
one should beware of the idea that a program can be adequately judged 
merely by a visual examination of its structural characteristics. There 
arc several reasons for this, (a) Firstly, there is scarcely a "theory” 
of what a maximally effective teaching program should look like, reliable 
enough to enable us to come to any definite contusions. We have a certain 
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body of general principles, but empirical investigations have given 
contradictory results on many points of detail, (b) Secondly, even if 
such principles were relatively clearly established, they cannot be a 
sufficient basis for an evaluation, since there are a whole series of 
other factors (e. g. organization of material) which affect effectivity. 

(c) Thirdly, even if the principles of programming were both clearly 
established and could in some way be shown to outweigh all other rea- 
sons for variation, a visual examination would still give insufficient in- 
formation about the value of a program, for the simple reason that many 
of these principles cannot be directly observed. A low percentage of 
error is an example of a programming variable which can only be de- 
termined by empirical methods. (The expert in a subject often guesses 
v/rongly about this. Precisely because of his expert knowledge, he con- 
siders the steps to be extremely ''small" and consequently estimates 
the percentage of error as "lower" than empirical methods show it to 
be. ) 

Instead of discussing whether the teaching material conforms to cer- 
tain programming principles, the experiment has been made of getting 
experienced teachers to judge its probable effectivity. One must, how- 
ever, allow for the fact that, because of lack of training in judging 
these new types of material, they can go seriously astray. In one study 
(Rothkopf, 1963) the results even showed a negative correlation (-0. 75) 
between teacher- estimated and real effectivity! 

To sum up, we can say, then, that while a visual examination is al- 
ways of interest, it must not be the only basis for the consumer's eva- 
luation of a teaching program. He must rely primarily upon how the pro- 
gram has been shown to work in practice, not upon what it looks like. 
Possibly, in the future, we shall Icnow so much more about the factors 
which produce an effective program that we shall not always reed to 
proceed via empirical demonstration. But in the present state of pro- 
gramming technique this is the only completely reliable way. 

When the consumer wishes to examine a program thoroughly, both 
visually and by studying the information given in the manual, it is ad- 
visable to use some form of check-list . This makes the task easier and 
also avoids a one-sided examination. The contents of the list can vary 
according to the needs of the particular teaching situation. We shall now 
give examples - partly based on an American "Checklist for selecting 
programs" (NSPI Journal, 1963) - of questions one often finds it 
necessary to put (cf. also e. g. Cox ct al. , 1962; Tracey, 1963; 
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Wark, 1963; and Vandcrschmidt , 1964): The question: arc so formu- 
lated that the answer '‘yee*^ usually gives the better alternative. In 
other words, a large number of positive answers is a better indicatioi^ 
than few positive answers. Hov/cver, some questions arc obviously 
more important than others. These must naturally be given the greatest 
weight in drawing the final conclusion. 

1 . Goal and curriculum ; 

1. 1, Does the publisher or constructor give a clearly formulated de- 
scription (in the form of observable behavior changes) of what the 
course is intended to teach? 

1. 2 Does a visual examination show that the description of the goal is 
fully covered by the course? 

1. 3 Does the goal of the programmed course (as it appears in points 

1. 1 and 1. 2) coincide in all essential respects with the curriculum: 
or general goal for your special teaching situation?' 

1.4, If there is any divergence, is it of such a character that it is pe- 
dagogically advantageous and would therefore bo accepted as a 
meaningful change in the course? 

1. 5. Is there reliable information about the time taken for the course 

by students of the type you meet in your teaching situation? 

1, 6, Will your students be able to get through the course, or as much 
of it as is desirable, in the time at your disposal? 

To sum up : Docs the general goal of the course sufficiently coincide 
v/ith the demands of your teaching situation? 

2. Student characteristics: 

2. 1, To get the best out of a course, certain skills and previous know- 
ledge are often needed. Are these ^‘starting requiremert s*' clearly 
described? 

2,2. Is there a pre-test, which can show whether a student has the 
required sldlls and loi owl edge to start the course? 

2, 3. Do these starting requirements coincide with what you could rea- 
sonably expect of a student in your teaching situation? 

2.4, Is there a pre-test which will show how much a student already 
laiows of the material the course is intended to teach? 

2.5, Is the course material so organized that the student with such 
previous knowledge can start the course at some other point than 
the beginning or otherwise go through the material in a way suited 
to liio special qualifications? 
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To sum up: Docs the general deeign of the course reasonably suit the 
group of students you arc teaching? 

3. Specific instructional effectivity of the study material : 

3-1. Is there a detailed and reliable report of the field try-out of the 

material in its finished state on representative groups of students 
(including pre- and post-tests)? 

3. Z. Arc the results of this field try-out satisfactory? 

3. 3. Were the conditions in the try-out sufficiently like those of your 
own teaching situation for you to be able to assume a similar re- 
sult in your own case? 

3. 4. Has a test been published with which you can check, by a satis- 

factory measuring procedure, whether the goal of the course has 
been achieved? 

3.5. Arc there grounds for assuming that the knov/lcdgc and skills 
acquired during the course arc retained for a reasonable time 
afterwards (i. e. has the try-out also studied retention after a cer- 
tain interval, v/ith satisfactory results)? 

To sum up : Arc there both (a) good grounds for supposing that the study 
material gives the desired specific results with typical groups of stu- 
dents and (b) the possibility of choc_s±ng in your ov/n teaching situation 
(suitable final tests)? 

4. General educational character of study material : 

4. 1. Docs the material give the student training in desirable study be- 
havior (c. g. does it involve more logical reasoning and induction 
from experience than copying and formal n'lcchanical working)? 

4. 2. Docs the material give the student sufficient opportunity for varied 
repetition (as against both insufficient and routine repetition)? 

4. 3. Docs the material give the student sufficient opportunity to inte- 
grate his knowledge (c. g. by problems which promote integration, 
work with supplementary and source material etc. )? 

4. 4. Is there a tcachcr^s handbook which gives examples of how to inte- 
grate the self-instructional material with other teaching methods? 
4. 5- Arc there interim tests built into or accompanying the course 

v/hich will be a good basis for possible additional help or teaching 
during the course? 

4. 6. Arc there gro\inds (as a result of the field try-out or research) 

for supposing that the material created an interest for the subject? 
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To sum up ; Has the study material such educationally desii'able 
characteristics that, in addition to its specific instructional effect, 
it also promotes such "fringe benefits" as good study technique, 
positive attitude to further study etc. ? 

5. Economic questions: 

5, 1. Is the cost of the complete material, including the necessary 

accessories such as teaching machines and/or printed forms for 
answers, covered by available resources? 

5. 2. Will it be easy for you to attend to the storage and upkeep of the 
material ? 

5. 3. If an expensive machine is necessary for the presentation of the 
material, will it be possible for you to use it for other courses? 
To sum uu : Is the purchase of the course material economically de- 
fensible ? 

In certain cases the consumer may consider it desirable to try 
out the material himself. This is perfectly legitimate, if the program 
is thought to fulfil all the consumer s^s requirements, but has been 
tried out on a group of students which differs essentially from those 
for whom the teacher intends to use it. In such a case, it is good 
strategy to try it out on a smaller group before the program is in- 
troduced on a large scale. 
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THE TERMINOLOGY OF PROGRAMMED IINiSTRUCTION AND EDUCA - 
TIONAL TECHNOLOGY: A SELECTIVE LIST 



Some terms within the area of educational technology and programmed 
instruction are listed with brief definitions. Naturally, complete coverage 
is not aimed at. Instead, some attempt has been made to include the 
major terms used in the present survey and/or that together form a rela- 
tively consistent frame of reference. 

Adaptive instructional system An instructional system which can adjust 
itself to varying student behavior by, for example, giving simpler 
tasks if the student has not been successful, and more difficxilt tasks 
when the student is successful. 

Algorithm General and non-ambiguous prescription for solving problems 
within a specific problem area. Part of the subject-matter analysis 
(as one step in the pre-writing phase when constructing self-instruc- 
tional study material) may be focused upon the questions: to what 
Q 2 ‘±cnt is it possible to formulate algorithms for the subject-area in- 
volved, and to what e:d:ent can and should these influence the instruc- 
tional sequence and method? 

Amplified T-tabulation A *’T -tabulation" (see that term) to which, be- 
sides the usual data from the goal analysis, has been added a list 
of items of information about the target population with point of de- 
parture from the student analysis. Reports on the special preliminary 
Icnowledge and existing skills of the students have then been added, as 
well as possibly a judgment as to how the ordinary occupation of the 
students after their specific educational training can affect the re- 
sults of the instructional program in the long run. An influence of 
this kind can r.aturally be both a support and a hindrance; hence, an 
appraisement in this respect can modify the programmer's treatment 
of the question of repetition and overlearning. (For illustration, see* 
chapter 6 above. ) 

Anschtltz diagram A diagrammatic method (developed by H. Anschtitz) 
for the evaluation and comparison of instructional programs, utilizing 
an examination of the distribution of concepts over the instructional 
micro-units. (For details, see Anschutz, 1965.) 
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Branching model A course layout which makes it possible for different 
categories of students (for example, students with different prepara- 
tory knowledge) to take separate paths thxough the study material. 

C-catalog A collection of fundamental concepts (C) in a certain subject 
area. Systematizing these logical '’basic units" within a hierarchica.1 
system results in a " structured C-catalog" . (A card file with the 
separate units on separate cards is often a practical set-up for such 
a catalog. ) 

C. C-catalog (or "CxC -catalog") A collection of instructionally interest- 
ing relationships between the logical "basic units" (C) witliin a cer- 
tain subject area. A catalog of this type (which in concrete form is 
ordinarily set up as a card file) can be made up with the help of a 
"C-matrix" (see that word). 

Chain model ("conversational chaining" ) A course layout in which the 
individual instructional units arc linked together in such a way that 
the feedback component of the preceding instructional unit is incor- 
porated as a part of the stimulc component of the following unit* In 
other words, the key answer is not given a separate presentation, 
but is embedded in the next unit, ordinarily as one or more words 
printed in italics. 

C-matrix An aid in mapping the instructionally interesting relationships 
between various fundamental concepts within a certain subject area. 

In principle it consists of a square- ruled diagram, the row and column 
headings of v/hich are the key words for the various "basic units" 
in the C-catalog, and every cell in the matrix is thought to symbolize 
all possible relationships between two such units. The constructor 
can, for example, go through the matrix cell by cell and ask himself 
questions of this type: What connection is there between these two 
concepts, and is this connection such that it can and should be used 
pedagogically? (For illustration, see chapter 7 above, ) 

Computer-assisted instruction Usually a special case of programmed 
instruction, viz, the case when a computer handles information 
storage, registrering , response comparison and feedback, during 
a highly individualized instructional program. 

Correlative stimuli Information, instructions tind evaluations, which in 
an instructional situation are presented in close connection with the 
principal points of the content but are not themselves meant to bo 
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learned. When these correlative stimuli have the chief function of 
facilitating the learning of certain other items, they can be called 
"prompts**. (Compare "Focal informative stimuli". ) 

Cor rule Sec "Didulc". 

Crowder model A course layout which combines a branched flow-model 
(in which the student working path is successively determined by 
his accomplishments) with use of multiple-choice responses. The 
model has been realized partly in book form (see the term "Scrambled 
book"), partly in the form of mechanical presentation (c.g. Auto- 
Tutor, Mark II). Crov/der^s most frequent use of the branching tech- 
nizuc seems to be for explanation of an error just made. 

Density of information A measure of the effective communication of 
information ('‘reduction of uncertainty" for the student; compare 
this word) in relation to some quantitative characteristic of the 
study material, c. g. , the number of instructional units. For example, 
tlie fewer "didulcs" (see that word!) which arc used to present a cer- 
tain amount of information to the student, the greater the density of 
information can be said to be. 

Didactogram A diagram (used during the empirical testing of instruc- 
tional programs) showing one or several students^ progress through 
a programmed study material. As background pattern for the didacto- 
gram, the previously worked out "didactoplan" may sometimes be 
used (sec word "Didactoplan"), after the necessary revisions and 
specifications have been incorporated. On such a p: :tcrn (showing 
the basic structure of the study material) arc presented the behavior 
data: in part, information as to the study routes used; in part, in- 
formation as to right and wrong performances. (For illustrations, 
sec chapter 14 above; especially Figures 14. 4 and 14. 5. ) 

Di dactoplan A graphic visualization (mainly made use of during the pre- 
writing phase) of the internal order in which one plans to bring up 
different subject-matter units (c. g. , concepts) in the auto-instruc- 
tional study material. By the use of special symbols, it can be shown 
on the .rt which points in the material will be gone through by all, 
and which will be presented only to certain students. 

Didule The fundamental instructional unit in programmed study mate- 
rial, ordinarily consisting of three components; the "stimule" (the 
information component), the "rcspule" (the response request), and 
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the ''corrulc^' (the feedback component). The didule can be said to 
represent the contribution of the program to the fundamental unit 
of interaction between the pupil and the program, which ordinarily 
has the temporal sequence: s-r-R-c. (The letters £, £ refer 

here to the three components mentioned above; capital R symbolizes 
the student^G answer reaction or "response''). - The didule is an 
instructional unit in programmed study material, that is, it is not 
identical with a presentation unit . (The most nearly equivalent ter n 
"frame" has the disadvantage of being used both for the concept 
’ instructional unit' and for the concept 'presentation unit' . ) 

Didule protocol An aid in the examination of an instructional program. 
The protocol consists of a chart in which each separate didule has 
a matching review column and the different review points are en- 
tered as row headings. (For illustrations, sec chapter 12 above, 
especially Figures 12. 2-12. 5. ) 

Educand The individual who is subjected to educational influence (in- 
struction, attitudinal influence etc. ). The "receiving" component in 
an educational process. In school, "educand" = "student", but 
"educand" is a more ^;cncral term. 

Educator Someone or something which causes educational influence 
(instruction, attitudinal influence etc. ). The "sending" component 
in an educational process. In school, "educator" often = "teacher", 
but "educator" is sometimes used in a more general sense, re- 
ferring also to non-personal sources of influence (self-instructional 
materials -and methods systems etc. ). 

Egrul Sec "Rulcg". 

Error rate The percentage of incorrect responses on a single instruc- 
tional unit or a set of such units when tested on a group of students. 
Dow error rate during work on program has often been considered 
desirable, but can also be obtained by non-desirable design (such as 
overprompting) and cannot therefore be used as an evaluation cri- 
terion by itself. 

Fading Successive diminishing of the number of elements in a stimulus 
complex. In programmed teaching the term is used in two contexts. 

(1) = Successive cutting down of prompts (so that the student gradually 
can give a correct answer without help, while he was able to give 
the correct answer in the beginning only with the help of certain clues; 
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’’fading of correlative stimuli”). - (2) = Successive disconnecting of 
separate parts of a larger stimulus complex, which is meant to be 
learned as a whole (’’fading of focal stimuli”). Example: The student 
learns a poem by repeated recitations. As a support for his memory 
he has a text for use in recital, which successively is cut down as 
words and sentences are deleted. - For meaning (2) tha term ’’vani- 
shing” is sometimes used. 

Feedback technique The interplay between the student and the study ma- 
terial, which tells the student how far he has succeded with each 
separate task. Ordinarily the feedback takes the form of a key an- 
swer presentation, but simple information as to whether the student 
response was right or wrong, or more detailed error explanations 
occur in some arrangements. It is typical of programmed instruction 
that feedback is almost immediate and very frequent. 

Filter didule A unit in the study material which informs us as to the 
best possible working track (branch) for the individual student. De- 
pending on what the more specific purpose is, we can spoa’.v of ”pre^ 
controlling” or ”i:>ost-controlling” didules (see these words). 

Focal informative stimuli The main points in the content which in an 
instructional situation are presented to a student to be ’’learned” 
(compare ’’Correlative stimuli”). 

Formal prompts Stimuli, which provide knowledge about the form of 
the expected response and thereby make it easier for the student to 
give the correct answer. The student can, for example, be allowed 
to see the first letter of the correct answer word or to know how 
many letters are in the word. (Since overuse of this type of clue 
technique can promote thoughtless answering with an inadequate 
direction of attention, it should as a rule be used sparingly. ) 

Goal adjustment Successive changes of earlier goal descriptions guided 
by empirical controls of the realism of the goal level (in concrete 
cases where the initial behavioral repertoire of the target group and 
certain resources are defined). Is sometimes considered to be the 
final phase of a complete goal-analytic process (cf. ’’Goal seeking” 
and ”Goal focusing”). 

Goal analysis Analysis of instructional objectives. A careful and de- 
tailed review of which knowledges and ckillo the students should bo 
able to display after going through a certain study material. (Compare 
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"Terminal behavior repertoire*'.) Sub-cornponents: "goal seeking", 
*'goal focusing", and "goal adjustment". 

Goal focusing Decisions about and formulation of the specific components 
of a goal description. "Goal focusing" can often be regarded as an 
intermediate link between "goal seeking" and **goal adjustment'* in 
a total goal- analytic procedure. 

Goal-related assessment See "Norm- related assessment". 

Goal seeking Collection of data as a basis for goal decisions and goal 
formulation. Normally, the goal seeking should involve empirical 
data collections, be as many-sided as possible, and also give starting- 
. points for priority ranking among various sub-goals. 

Horizontal panel-book A programmed textbook in which the pages are 
divided into panels which are not read in the usual reading sequence 
(from top to bottom) but from page to page (or from sheet to sheet). 

The student reads, in other words, the top part of the pages through 
the whole book (or a large part of the book) before he goes over to 
another section of the same pages. The key answer for a given t«ask 
does not usually appear in the same section or on the same page, but 
separately on the back of the section or in connection with the next 
information unit. 

Instructional effectivity Some measure of the instructional result in 
relation partly to a specific goal, partly to instructional time ex- 
pended, An effective instructional situation is, therefore, a situation 
in which an optimal balancing of behavioral products (as defined by 
the goal analysis) is achieved with the minimum expenditure of time. 

Integrator A teaching aid which offers the student a constantly alter - 
nating sequence of (a) pre sentation of material and (b) the possibility 
of response reaction. Programmed textbooks and teaching machines 
are typical examples. An integrator, through this double function, is 
different from the much more usual aids in conventional teaching, 
which either have just the function of "presenting" or^ only give the 
only give the student an opportunity for a response reaction. The 
term ** presentatoi" * describes an aid in teaching, the chief task of 
which is to present to the student a stimulus - sequence prepared in 
advance (an ordinary textbook or educational film would be typical 
examples). The term " reactor, on the other hand, refers to a 
teaching aid, the chief function of which is to offer the student 
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opportunities to give response reactions. (Example: Question lists. ) 

Intermittent model A layout of the auto -instructional material which 

demands a fairly independent treatment of the material by the student. 
He is presented with a *‘guide -book” containing a detailed study plan 
and self-correcting part tests, but the main material is collected by 
the student from different sources outside the guide-book. If the re- 
sult of a test is good, the student is directed to new material. If it 
is not good, he is sent back to the same source (or one parallel to it). 
This shifting between instructions, independent study, and diagnostic 
review tests places the intermittent model somewhere between a 
completely programmed course and Pressey^'s ”tail model” (see that 
word). 

Ite / it ion model A course layout in which the student alv/ays goes through 
the various instructional micro-units in a pre-established order, the 
same for all students, but where he is directed to repeat single units 
(or groups of units) under certain achievement conditions. 

Linear model A course layout with the same task sequence for all stu- 
dents. 

Matlietics A systematic approach to the analysis of instructional compo- 
nents and the construction of instructional units and sequences, de- 
vised by T. Gilbert. 

Micro-branching model A ‘‘branching model” in which a certain amount 
of student differentiation may take place within the separate instruc- 
tional unit (c. g- , different error explanation), v/hilc these units as 
a whole arc taken in the same order by all students. Most scrambled 
books, for instance, are of this type. (See ’’Branching model” and 
“Scrambled books”. ) 

Multiple success strategy An attempt to create a dual motivation 

(possibly for different categories of students simultaneously). This 
can mean, for cxam^ole, that on the one hand the student works 
through the study material with a high probability of getting the correct 
answer to separ -cc tasks (“sure winner”), while on the other hand he 
is offered “extra success possibilities” (for example, possibilities 
for fast-track procedure and/or the satisfaction of answering correct- 
ly without clues, ’’individualized clue feeding”). 
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Norm~related assessment An assessment of the instructional result in 
which the single student^s result is related to information about 
other students'" results. (For example: *‘C^n this test, student A has 
obtained a result which places him in the best Z5 % of group G. *') 

In contrast, ’“goal-related assessments” relate the instructional re- 
sult to a specific scries of goal descriptions. (For example: ”Out of 
the items of this test, designed to cover all relevant goal descriptions, 
student B has mastered 80 %”. ) 

Objective A statement describing the behavior the student is expected 
to acquire from a course. 

Parallel track arrangement A branching model (see that word) which 
allows two or more separate categories of students to go through 
differently designed study plans with the same final goal. The same 
principle or group of problems can, for example, be treated from 
various points of view, depending on the varying preliminary attitudes 
of the different students. 



Passage criterion The lowest possible point which, the student can be 
allowed to reach with reference to a certain terminal behavior. In 
goal analysis it is important to fix specific values of this kind, as 
soon as v/e are dealing with a continuum of skill within a certain area 
of behavior. Without a clearly defined "r'iinimum for acceptability” 
it is impossible to judge how well the goal set by the course has been 
reached. The passage criteria can suitably be included in the ”T- 
tabulations”. (Compare ’=Goal analyse s”^ ” Terminal behavior”, and 
”T-tabulation”. ) 

Post-controlling didule A unit in the study material which attempts to 
ascertain whether the student has learned carefully enough what has 
just been studied. If his responses show that this is not the case, the 
student is directed back tc a .section which has already been gone 
through or to an alternative repeat section (with similar, although 
not identical content). 

Pre-controlling^ didule A unit in the study material with which an attempt 
is made to determine whether the student has sufficient preparatory 
Icnowledge to be able to skip a certain section of the course. 



Pro sentator See ‘‘Integrator 



Programming Construction of auto-instructional study-material accord- 
^ ing to a three-phase procedure consisting of (a) preparatory work 
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(*‘prc -writing’* phase) \vith, among other things, emphasis on goal 
analysis and subject-matter analysis, (b) composition of a prelimin- 
ary version utilizing certvain psychological principles of learning and 
(c) a ’’post- writing j^hasc”, including successive empirical try-outs 
and revisions in the preliminary versions until the goal determined 
in advance has been reached. 

Prompt Some stimulus added to the terminal stimulus to make a correct 
response more likely during a learning process. Ax^proximatc syno- 
nyms (at least in some texts): Cue a.nd hint- Sometimes two major 
types arc d||^inguishcd: “Formal prompts’* and “Thematic x-^rompts” 

(sec these words). 

Reactor Sec “Integrator". 

Reduction of uncertainty A measure of the average effectivity of an 
instructional material, expressed as a reduction of the number of 
errors (within a specific student group). If we examine the ability 
of the students to solve a related terminal test item before going 
through the material ("error percentage A"), as well as their ability 
to solve the problem after going through the material ("error percentage 
B"), then the measure of reduction of \incertainty would be "error per- 
centage A - error percentage B". (Compare terms "Speed of informa- 
tion" and "Density of information". ) 

Rcs]Donsc request That part of an instructional micro-unit which pre- 
sents the student with the task of giving an active response reaction 
(to solve a problem in mathematics, to fill in a word which has been 
left blank, choose the right alternative from several which have been 
presented, etc. ). It is typical of the programmed instruction that 
these response requests are very frequent and continually alternate 
with the presentation of new material. 

Respulc Sec "Didulc". 

Reverse accumulation A system for practising a temporal behavior 
sequence as follows: the final phase of the series is practised first, 
after this the next to the last plus the, la.st , etc. Besides the fact that 
the student practises 'Sequences (instead of separate phases only), the 
advantage of this axirangement is that the student cxxDC.riences a mo- 
tivation in being able, at an early stage, to perform the final phase 
in a behavior series. This is especially true if this final phase has 
the character of a desirable goal. 
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Ruleg ( 1 ) A sequence in programmed study material in which new in- 
'rmation is introduced according to the formula: first general rule 
(law, principle, definition or the like), then a comjilete example, 
then an incomplete example for the student to v/ork with. That is, 
the '*ru'* (rule) precedes the ”cg's’‘ (examples). ( 2 ) The term is also 
used (and originally) for a special system of constructing sclf- 
instructional study material, developed by Evans, Homme (k Glaser 
(1962), in which the use of the sequence type metntioned above is an 
important part, but that also has a series of other typical character- 
istics. - The term '‘EgruP' is sometimes used for the more inductive 
sequence (leading the student through some examples before letting 
him try to formulate the rule h rnself); hence, the opposite of Ruleg 

( 0 - 

Scrambled book A programmed textbook characterized by branches in- 
corporated for the purpose of explaining errors. After going through 
the first information unit, the student must answer a multiple -choice 
question as control. The answer he chooses determines the page to 
which he is directed next. If he has chosen the right ^nswer, the nev/ 
page presents new information and a new task. If his choice has 
been wrong, on the other hand, he is given an explanation of his error 
on the new page and as a rule is directed back to the original task 
page with the instruction to choose a better answer. 

Skinner model A course layout which combines a non-branching flow 
model (linear model where all students go through the same study 
material in the same order) with a demand for self-constructed reciDon- 
ses. The name derives from B. F. Skinner but is somewhat mis- 
leading, since Sldnner worked with other models also (among them 
models with extreme individualization of the repetition procedure). 

Speed of information A measure of the effective communication of in- 
formation (''reduction of uncertainty" for the student; compare this 
word) in relation to the period of time needed for learning. A measure 
which indicates, in other words, how much effective information the 
students assimilate per time unit in going through a certain instruc- 
tional program, and which thereby gives us one meaningful point of 
departure in evaluating the effectivity of the program. 

Spiral progression A sequence of material set up in such a way that the 
student goes through different subject-matter areas at a lower grade 
O of difficulty before going on to these different areas of material at a 
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higher level of diffi::ulty. 

Stimulc See **Didule*‘ 

Student ~pa :ed flow of information The fact that in programmed instruc- 
tion the student himself most often decides when a new information 
unit shall be presented to him (as opposed to lessen and lecture 
teaching where the flow of information is directed by the teacher). 

Surplus learning An extra dividend in the learning result, over and 

above the specific luiov/lcdges and skills. Acquired study techniques, 
attitudes to a certain field of knowledge and the like may be considered 
’’surplus learning”. 

System analysis A general term for various types of studies on the 
different components of a system. A system analysis is usually a 
very important preparatory phase of work in constructing an instruc- 
tional methods -and-materials system and then includes such things 
as goal analysis, student analysis, situation analysis, subject-matter 
analysis and media analysis. 

System synthesis A general term for putting together the components 
of a system. In constructing a methods-and-materials system for 
instruction, v/e usually start with an analytic phase (studying the pre- 
requisites one by one) and follow up with a system synthesis in con- 
structing a preliminary version. 

’’Tail model” A layout of auto -instructional mat eric'll which presupposes 
that the student has already in some other way acquired the main 
points of the study material (for instance, in an ordinary textbook). 

The auto-instructional material functions then as post- study control 
and remedial teaching where needed. The foremost advocate of this 
partial type of auto -instruction has been S. L. Pressey. 

Target population That group of students to which a certain study mate., 
rial is directed. 

Terminal behavior or terminal behavior repertoire The Icnov/ledge and 
skill which the student should be able to display after going through 
a certain study material. A careful mapping of the desired terminal 
behavior is an important part of the preliminary work in drawing up 
a programmea course. (Compare ’’Goal analysis”. ) 

Terminal stimulus The unprompted question or i^roblem to which the 
Q student is taught to respond. (Compare the word ’’prompt”. ) 
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Terminal test An examination which makes it possible to ascertain 
whether a student who has gone through a given study material can 
demonctrate the loiowledge and skill which arc specified in the goal 
description of the course, (Also: ''criterion test". ) 

Thematic ^prompts Stimuli which with the help of meaningful (content - 
related) associations make it easier for the student to give the 
correct answer. Examples and logical deductions can be counted 
as belonging to this group of hints. 

T -tab ulation A tabulation (used as an aid in goal analysis), with the 
help of which the programmer gets a survey of (a) the specific "ter- 
minal behaviors" (sec that word) v/hich the student should display 
at the end of the course as well as (b) under which detailed conditions 
this is to take place. The T-tabulation consists of a cross-classific- 
ation chart of specific terminal behaviors on the one side and a list 
of "specifications" (which describe approved aids, acceptable mini- 
mum speed, maximum tolerance of error etc, ) on the other, (For 
illustration, see chapter 5 above. ) 

Tut of or A general term for an aid in presenting a teaching program to 
a student. This aid, or tool, can take the form of a printed book, 
which we then refer to as a "programmed textbook" (or "tutoprint"). 
But it can also be of mechanical character, and we then S 2 :>eak of a 
"teaching machine" (or "tutomat"). 

Vertic al panel-book A programmed textbook, in which the pages are 
divided into panels which (in contrast to the case of the "horizontal 
panel -book", compare that word) are read in the usual reading se- 
quence (from top to bottom of the page). Each panel usually contains 
a short information unit to which is linked a response request. A 
key answer is also usually part of the instructional micro -unit, and 
this can be placed in the same panel ("simple marginal-key"), in 
connection with a later panel ("delayed marginal-key"), or also on 
the back of the information page, 

Weltner technique A technique for assessment of the instructional 

effect of a particular program, utilizing the difference in subjective 
information of a basic text before and after the instruction. The 
difference is established by means of a sx^ecific technique of text 
predictions, similar to one introduced by C, E. Sliannon, (For de- 
tails, see Weltner, 1966.) 
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Zig-.zag book A j>rogrammed textbook in which the pages are divided 
into cut sections which can be turned forward and backward inde- 
pendently of each other. This provides a better stimulus situation 
than that offered by the ordinary forms of horizontal and vertical 
panel-books. That is, the key answer is not in the visual field when 
the student answers the problem, but both the i^roblem and the key 
answer are in the visual field after the student has answered. (For 
illustration, see chapter 8 above, especially Figures 8. 7 and 8. 8. ) 



109 



o 

ERIC 



20. I - 



PROGRAMMED INSTRUCTION AND EDUCATIONAL TECHNOLOGY : 
A SELECTIVE BIBLIOGRAPHIC GUIDE 



Introductory note 

The present bibliographic guide is selective, that is, it does not at all 
aim Bit being comprehensive. Instead, we have tried to cover a number of 
basic topics and, within each of these, to list a sample of pertinent 
references. For some of the categories we have also attempted to give 
some further guidance by means of brief comments. 



Outline of contents 



1 . 

1 . 1 
1 . 2 

1. 3 
1.4 

2 . 

2 . 1 
2 . 2 

3. 

3. 1 



3. 2 



3. 3 



3.4 

3.5 

3.6 



Survey publications 
Integrated surveys in book form 
Survey articles 

Introductory films and film strips 
Inventory- type surveys 

Collections 

Collections of articles in book form 

Journals and bulletins with continuous interest in programmed 
instruction and educational technology 

Special-topic publications 

The pre-writing process: Introductory analyses and decisions 
("System analysis") 

3.1.1 Analyses of terminal objectives, target populations, and 
subject-matter structure 

3.1.2 Decisions on presentation media 

Problems related to the writing process ("System synthesis") 

3. 2. 1 The problem of "flow structure" 

3. 2. 2 The characteristics of the single educational units 
The post-writing process ("System modification and evaluation"): 
Supplementary examination, step-by-step try-outs, revision, and 
evaluation ordinal program products 
Theoretical discussions 

Programming activities in various countries 

From the discussions and research on programmed instruction 
in Sweden 
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1. SURVEY PUBLICATIONS 



1 . 1 Integrated Surveys in Book Form 

Becker, J. L. A programed guide to writing auto-instructional programs . 
Camden, N . J . : RCA , 1963. 

Bcrnmalm, S. Programmerad undervisning: En granskning och samma.n - 
stailning av alne rikanska forGkningsrapporter 7 Gbteborg: IPU, 1965. 

Bjerstedt, A. Programmerad undervisning, sprS^klaboratorier och grup p- ^ 
dynamisk kaVtl&ggning . (4th ed. ) Lund: Gleerupo, 1968. 

Bjerstedt, A. Sj Alvins true rande studiemate rial . (Rev. ed.) Lund: Uniskol, 

1969. 

Brethower, D. M. Programed instruction: a manual of programing 
techniques . Chicago: Educational Methods, 1963. 

Brethower, D. M. Programed instruction: a manual of programing 
techniques . A rbor, Mich. : Ann Arbor Publisher^, 1965. 

Dodd, B. Programmed instruction for industrial training. London: 
Heinemann, 1967 . 

Espich, J.E. & Williams, B. Developing programmed instructional 
materials. Palo Alto, Calif. : Fearon, 1967. 

Filby, Y. Teaching machines. A review of theory and research. Copenhagen 
Munksgaard, 1961. 

Fry, E. B. Teaching machines and programmed instruction. An introduction . 
New York: McGraw-Hill, 1963. 

Garner, W. L. Programed instruction. New York: Center for Applied 
Research in Education, 1966. 

Green, E. J. The learning process and programmed instruction . New York: 
Holt, 1962. 

Hughes, J. L. Programed instruction for schools and industry . Chicago: 
Science Research Associates, 1962. 

Kay, H. , Dodd, B. & Sime, M. Teaching machines and programmed 
instruction. Harmondsworth: Penguin Books, 1968. 

Leedham, J. & Unwin, D. Programmed learning in schools. London: 
Longmans, 1965. 

Leith, G. O. M. et al. A handbook of programmed learning. (Rev. ed. ) 
Birmingham, England: University uf Birmingham, 1966. 

Leith, G. O. M. Seconds thoughts on programmed learning. London: NCET, 
1969. 

Lysaught, J. P. & Williams, C.M. A guide to programmed instruction . 

New York: V/iley, 1963. ^ 

Markle, G.M. , Eigen, L. D. Komoski, P. K. A programed primer on 
programing. Vol. I. Principles . New York: Center for Programed 
Instruction, 196 1 . 

Markle, S.M. A programed primer on programing. Vol. II. Practical 
problems . Nev/ York: Center for programed Instruction, 1961. 
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Marklc, S.M. Good frames and bad . (Rev. ed. ) New York: Wiley, 1969. 

Ofiesh, G. D. Programed instruction: A guide for manap;cment . New York: 
Amcr. Management Ass. , 1965. 

Rowntrcc, D. Basically branching. A liandbook for programmers . London: 
MacDonald, 1966. 

Schicfclc, H. Huber, G. L. Programmie rte Unterweisung - 
programmiert. Miinchen: Elircnwirth, 1969. 

Schramm, W. Programed instruction today and tomorrow . New York: 

The Fund for the Advancement of Education, 1 962. 

Shirley “Smith, K. Programmed learning in integrated industrial training. 
London: Gower Pr. , 1963. 

Silvern, L. C. Fundamentals of teaching machine and programmed 

learning systems. Los Angeles: Education and Training Consultants, 
1964. 

Skinner, B.F. The technology of teaching. New York: Appleton- Century- 
Crofts, 1968. 

Stiernborg, M. Programmerad undervisning - analys och bversikt . 
Stocldiolm: Dept. Educ. , 1966. 

Stolurow, L. M. Teaching by machine . Washington: U.C. Government 
Printing Office, 1961. 

Taber, J.I. , Glaser, R. & Schaefer, H. H. Learning and programmed 
instruction . Reading, Mass. : Addison- Wesley, 1965. 

Thomas, C.A. eta?.. Programmed learning in perspective. A guide to 
programme writing . Barking, Essex: Adelphi, 1963. 

Undervisningsmaskiner och programmerat studiematerial . Kungl. Skol- 
bvcrstyrelsen: Utredningar i skolfrigor 11, 1963. 

Zielinski, J. Scholer, W. Methodik des programmierten Unterrichts . 
Ratingen: Henn, 1965. 

Ziclke, \7. Programmie rte Instruktion in der Wirtschaft. Munch en: 

Verlag Moderne Industrie, 1970. 

Note : Some of these books are very elementary and popular in style; 
whereas others - for instance, the publicatioro by Filby and Stolurow - 
arc more adapted to readers witli some familiarity of the terminology 
and style of scientific psychology. In some cases the presentation form 
is that of a programmed textbook and, hence, functions both as introduction 
and illustration (for instance, Becker, 1963; Brethower, 1963; Markle 
et al. , 1961; Schiefele & Huber, 1969). Those who arc most interested 
in the program construction process will find Fry (1963), Lysaught 
Williams (1963), and Marklc (1969) to be some of the important references. 
The aspects of learning theory are given special emphasis by Green (1962). 

1 . 2 Survey Articles 

Holland, J. G. Research on programming variables. In: R. Glaser (Ed.), 
Teaching machines and programmed learning, II. Washington, D. C. : 
NEA, 1965. Pp. 66-117. 

Klaus, D. J. An analysis of programming techniques. In: R. Glaser (Ed.), 
Teaching machines and programmed learning, II. Washington, D. C. : 
NEA, 1965 ." Pp. 118-161. 
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Lcib, J. W. et al. Teaching machines and programmed instruction. 

Psychol. Bull. , 1967, 12-26. 

Lumsdainc, A. A. Instruments and media of instruction. In: N. L. Gage 
(Ed. ), Handbook of research on teaching . Chicago: Rand McNally, 

1963. Pp. 583-682. 

Morrill, C.S. Teaching machines : A review. Psychol. Bull., 1961, 
363-375. 

Skinner, B. F. Why we need teaching machines. Harvard cduc. Rev. , 

1961, 377-398. 

Skinner, B.F. Reflections on a decade of teaching machines. In: 

R. Glaser (Ed. ), Teaching machines and programmed learning, II. 
Washington, D. C. : NEA, 1965. Pp 5-20. 

Spaulding, S. Advanced educational technologies. Prospects m Education 
(Uncsco), 1970, (3), 9-19. 

1 . 3 Introductory Films and Filmoirips 

**Examplc of a teaching machine program ^\ Filmstrip prepared by 
D. H. Luxton & R. E. Corrigan. Color, 69 frames. Pasadena, 

^ alif. : Basic Skill Films. 

^'Learning and behavior '^ Interviews with B. F. Skinner and R. J. 

Hcrrnstcin at Harvard's Psychological Laboratory (CBS). B & w, 

16 mm sound, 27 min. New York: Carousel Films. 

**Onc step at a time ^h Produced by D. Klaus et al. , American Institute 
for Research. Color, l 6 mm sound, 28 miu. Pittsburgh, Penna. : 

AIR. 

^^Programming is a process **. Filmstrip (color, 80 frames) and sound 
tape. Produced by S.M. Marklc and Ph. W. Tiemann. Chicago: 

Tiemann Associates. 

^^Sclcction and use of programmed materials **. Filmstrip (color, 64 
frames) and sound record, 16 min. Washington: NEA. 

*^Tcaching machines^* . Filmstrip prepared by W. H. Allen. Tcchn. 
advisor: A. A. Lumsdainc. Color, 62 frames. Pasadena, Calif.: 

Basic Skill Films. 

**Tcaching machines and programmed learning^ h Demonstrations and 
comments by B.F. Skinner, A. A, Lumsdaine, and R. Glaser (NEA). 

B & w, 16 mm sound, 29 min. "Washington: Norwood Films. 

1 . 4 Inventory-Type Surveys 

Note : Many bibliographical surveys have appeared, but they too often 
have the disadvantage of becoming obsolescent almost before leaving 
the printing press, due to the rapid increase in literature. Only some 
examples arc therefore listed here. For information about the most 
recent publications, the reader should consult the latest issues of the 
journals in the field, where surveys and reviews often appear. - As is 
apparent from the titles, the inventories cover different things. Inventories 
of machines (not quite up to date any longer, of course) arc given by 
Finn & Perrin and by Ross, whereas program inventories arc presented 
in ‘^Programs, '62”, ’^Programs, '63” etc. and in Hendershot, Finished 
research is presented in Schramm. A more general bibliographic aim 
is represented by Gee. 

Bjerotedt, A. The terminology of programmed instruction. Didakometry , 
No. 13, I 766 . 
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Cavanagh, P. & Jones, C. (Eds.) Programmes in print lVo6. London: 
Association for Programmed Learning, 1 9^^ 

Cavanagh, P. & Jones, C. (Eds. ) Yearbook of educational and instructional 
technology 1969/70 . London: Cornmarket, 1969 . 

Center for Programed Instruction. Programs, *62. A guide to programed 
instructional materials available to educators by September 1962. 
V/ashington: U.S. Government Printing Office, 1962. 

Center for Programed Instruction. Programs, *63. A guide to programed 
instructional materials available to educr ^^o rs by September 1963. 
Washington: U.S. Government Printing Office, 1963. 

Center for Programed Instruction. Programed instruction materials 
1964- *65. A guide to programed instruction materials available for 
use in elementary and secondary schools as of April, 1965. New York: 

CPI, Columbia Univer. , 1965. 

Centre de Documentation sur I'Enseignement Programme. Bibliographic 
sur l*cnGcignement programme et les machines a enseigner. I-Il. 

Paris: Institut Pedagogique National, 1966. 

Centre dc Documentation sur I'Enseignement Programme. Catalogue 

analytique des cours programm es. (Languc franqaise.) Paris: Institut 
Pedagogique National, 1 967 . 

Committee for Out-of- school Education, Council for Cultural Co-operation. 
Programmed instruction: Institutions and their activities . Strasbourg: 
Council of Europe, 1970. 

Deutschsprachige Lehrprogramme , Stand: Mai 1969. Berlin: P^dagogisches 
Zentrum, 1969. 

Finn, J. D. & Perrin, D. G. Teaching machines and programed learning . 

A survey of the industry. V/ashington: U.S. Government Printing Office, 
1962. 

Ge e , R . D . Teaching machines and programmed learning. A guide to the 
literature . Hatfield: HERTIG, 1965. 

Gesellschaft fxir programmiei'te Instruktion. Kontakt-Registe r der padago- 
gischen Technik . (3. Aufl. ) Quickborn: Schnelle, 1970. 

Hendershot, C.H. Programmed learning: A bibliography of programs and 
presentation devices . (4tli ed. & suppl. 1-6. ) Bay City, Mich. : Author, 
1970. 

Hintermaier, R. Ed. ) Le luiprogramme **68. Mxlnchen: Ehrenwirth, 1968. 

Hintermaier, R. (Ed.) Le rnprogramme **70. Munchen: Ehrenwirth, 1970. 

Mxil.ier, D. D. Jahrcskatalog: Kybernctik, Automation, Programmierter 
Unterricht, Grenzgebie te . Berlin: Elwert und Meurer, 1968. 

National Society for Programmed Instruction. Directory of members . 

San Antonio, Texas: NSPI, 1971. 

Programmed instruction guide. (2nd cd. ) Nev/buryport, Mass. : Entelek, 

1968 . 

Programmed learning and teaching machines: Bibliographical references. 

Antwerpen: international Audio-Visual Technical Centre. (Undated issues. ) 

Ross, V/. L. m. fl. Teaching machines: Industry survey and buyers^ guide. 
New York: Center for Programed Instruction, 1962. 



114 



- 20, 6 - 



Schramm, V/, The research on propcramcd instruction . V^aohington 
Government Printing Office, 1964. 

Spaulding, S, Programmed in struction: An international directory, 
Uncaco, 1967, 
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2. COLLECTIONS 

2 . I Collections of Article g i n Book Form 

Aagaard, K. , Dohn, H. & Marckmanii, W. (Eds. ) Konference om program" 
merct undervisning . Copenhagen: Lciore rforeningernes Mate rialudvalg, 
1968 . 

Bung, K. (Ed.) Programmed learning and the language laboratory (l). 
London: Longmac, 1968. 

Coulson, J.E. (Ed. ) Programmed learning and computer-based instruction. 
New York: Wiley, 1962. 

Dalillof, U. Wallin, E. (Edc.) Liiromedelsfor skning och unde rvisnings - 
planerinK * Stockholm: Nord. ut r e dnin^ ss c r ic , 1 9^9. 

DcCecco, J. P, (Ed. ) Educational technology. Readings in programmed 
instruction . New York: Holt, 1964. 

Dolmatcli, Th. B. m. fl. (Eds. ) Revolution in training. Programed instruction 
in industry. New York: American Management Acs. , 1962. 

Dunn, V/.R. & Holroyd, L. (Eds. ) Aspects of educational technology . 

Vol, 2. London: Methuen, 1969. 

Filep, R. T. (Ed. ) ProGpectiv;G in programing. New York: Macmillan, 

1963. 

Frank, H. (Ed. ) LehrmaGcliinen in kybernetischer und p^dagogische r 
Sicht. 1-4. Miinchen: Oldenbourg, 1963-66. 

Galantcr, E. (Ed. ) Automatic teaching: The state of the art. New York: 
V/iley, 1959. 

Gcsellschaft fur Programmierte Inctruktion. 8. Internationales Symposion 
liber Programmie rte InGtrulction und Lehrmaschinen; Kur zfas sungen> 
Berlin: GPI, 1970. 

GlaGer, R. (Ed. ) Teaching machines and programmed learning, II: Data 
and directions. Washington: NEA, 1965. 

Goldsmith, M. (Ed. ) Mechanisation in the classroom. An introduction to 
teaching machines and programmed learning . London: Souvenir Pr. , 

1963. 

Goodman, E. (Ed. ) Automated education handbook . Detroit, Mich.: 
Automated Education Center, 1965. 

Hughes, J. L. (Ed. ) Programed learning: A critical evaluation . Chicago: 
Educational Methods, 1963. 

Lambert, Ph. (Ed.) The teacher and the machine. Madison, Wise.: 

Dcmbar, 1962. 

La recherche en enseignement programme . Paris: Dunod, 1969. 

Lehnert, U. (Ed. ) Elektronischc Datenve rarbeitung in Schule und Aus- 
bildung . Miinchen: Oldenbourg, 1970. 

Lumsdaine, A. A. (Ed.) Student response in programmed instruction . 
V/achington: National Academy of Sciences, 1961. 

Lumsdaine, A. A. Glaser, R. (Eds.) Teaching machines and programmed 
learning: A source book . V/aohington: NEA, 1960. 
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Lys aught, J. P. (Ed. ) Programmed learning, Evolvin;; principles and 

industrial applications . Ann Arbor, Mich. : The Foundation for Research 
on Human Behavior, 1961. 

Mann, A. P. Bruns trom, C.K. (Eds.) Aspects cf edicational technology . 
Vol. 3. London: Pitman, 1969^ 

Margulics, S. & Eigen, L. D. (Eds. ) A pplied programed instruction . 

New York: Wiley, 1962. 

Ofiech, G. D. & Meierhenry, W. C. (Eds.) Trends in programmed 
instruction. Washington: NEA, 1964. 

Praxis und Perspektiven dcs programmie rten Untcrrichts . (Referatc dcs 
3. Nfirtingcr Symposions fiber Lehrmaschinen. ) Quickborn: Schnelle, 
1965. 

Praxis und Perspektiven des programmierten Unterrichts, Band II . 

Referate des V. Symposions fiber Lehrmaschinen.) Huickborn: Schnelle, 
1967. 

Programmierter Unterricht und Lehrmaschinen. Bericht. Internationale 
Konferenz . Berlin: P^dag. Zentrum, 1964. 

Rollett, B. (Ed.) Praxis und Tlieorie des programmie rten Unterrichtes . 
Stuttgart: Klett, 1970. 

Rollett, B. & Weltner, K. (Eds.) Perspektiven des programmiertes 
Unterrichts , Wien: Osterreichischer Bundesverlag, 1970. 

Rollett, B. & Weltner, K. Fortschritte und Ergebnisse der Unter- 
richts technologic. Mfinchen: Ehrenwirth, 1971. 

Roucck, J.S. (Ed.) Programmed teaching . New York: Philosophical 
Library, 1965. 

Schestakow, A. W. (Ed,) Programmiertes Lernen und Lehrmaschinen . 
(Transl. fr. Russian.) Berlin: VEB Yerlag Technik, 1965. 

Smith, V7.I. & Moore, J.Yl. (Eds.) Programmed learning: Theory and 
research . Princeton, N. J. ; Nostrand, 1962 

Teal, G. E. (Ed. ) Programmed instruction in industry and education . 
Stamford, Conn7 : I^blic Service Research, 1963. 

Tobin, M. J, (Ed. ) Problems and methods in programmed learning, 1-5 . 
Birmingham, England: The National Centre for Programmed 
Learning, 1967. 

Unwin, D. & Leedham, J. (Eds.) Aspects of educational technology . 

Vol. 1. London; Methuen, 1967. 

Yana, J. & Tollingerova, D. (Eds.) Programovanc uceni . (Programmed 
learning and teaching machines, I-II. From the Liblice Conference, 
1965. ) Praha: Pedagogicky ustav JAK CSAV, 1966. 

Wallin, E. (Ed. ) Undervisnihg - konst eller teknik? Stockholm: Almqvist 
V/iksell, 1968 ^ 

Zifreund, V/. (Ed.) Schulmodelle, programmierte InGtruktioncn und tech- 
nisclie Medien. Mfinchen: Ehrenwirth, 1968. 

Note : Among the many collections of articles available, tlic early one 
by Lumdaine & Glaser (I960) is probably still one of the best sources 
for anyone who wants to have a survey of the brief history of programmed 
instruction and to read some of the most important early articles by the 
originators in the field: Pressey, Skinner, Crowder, and others. This 
book is also a good bibliographic source for ’^old“ publications (up to I960). 
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A continuation volume, containing among other thii.'.gG a retrospective 
essay by B.IT. Skinner, is Glaser (1965). The development in Great 
Britain can be followed in, for example, Unwin & Leedham (1967), Dunn 
& Holroyd (1969), and Mann & Brunstrom (1969). The development in 
German- speaking countries is mirrored in the many reports from the 
symposia of Gesellschaft filr programmie rte Instruktion, for example: 

Frank, 1963-66; Praxis und Perspektiven des programmierten Unterrichts, 
1965 and 1967; Rollett, 1970; Rollett & Weltner, 1970 and 1971. 

2. 2 Journals and Bulletin s 

AID: Auto-Instructional Devices . Has been published by Institute of 
International Research and Development, Lubbock, Texas. 

(Cf. NSFI Journal below) 

Audiovisual Instruction. Ten issues per year. Department of Audiovisual 
Instruction, NEA, Washington. 

Automated Education Letter . Monthly (since 1965). Automated Education 

Center, Detroit! (Supplements E.H. Goodman (Ed.), Automated Education 
Handbook. Detroit: AE Center, 1965.) 

AV Communication Reviev/ . Quarterly. Department of Audiovisual 
Instruction, NEA, Washington. 

Bibliographic: Pro,n;rammic rte r Untcrricht . Berlin: P^dagogisches 
Zentrum. 

Deutsche Lehrprogramme filr Schule und Praxis . Has been published 
quarterly by Manz Verlag, Miinchon, Germany. 

Didakometry . Mimeographed bulletin. Department of Educational and 
Psychological Research, School of Education, Malm5, Sweden. 

Educational Technology . (Formerly: Teaching Aids News.) Monthly. 
Educational News Service, Saddle Brook, New Jersey. 

Enscignement Programme . Quarterly. Published by Centre de Documentation 
sur l^Ens eignement Programme, Paris. 

Journal of Educational Teclinology . Published three times a year (since 1970) 
by tlie National Council for Edu cational Technology; England. 

Journal of Programmed Instruction. Quarterly. Has been published by 
Center for Programmed Instruction, Teachers College, Columbia 
University, New York. 

La Cybernctique ct la Pedagogie Cybe rnetique . L^ As sociation de 
Pedagogie Cybe rnetique, Paris. 

L^Enseignement Programme . Quarterly. Institut Pedagogique National, 

Pa r i G . 

NSPI Journal . Ten issues per year. The National Society for Programmed 
Instruction, USA. A combination of AID (cf. above) and the earlier 
NSPI Newsletter. 

Programmed Instruction . Nine issucc per year. Has been published by 
Center for Programmed Instruction, Teachers College, Columbia 
University, New York. 

Programmed Learning and Educational Technology . Association for 
Programmed Learning, London. 

Programmie rtes Lerncn, Untcrrichtstechnologie und Unterrichtsforschung . 
Quarterly. Cornelsen, Berlin. 
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Visual Education (incorporating ’^ Programmed Learning Ncws' Q Monthly. 
London: National Committee for Audio-Visual Aids in Education. 

Zentralblatt der Gcsellschaft fiir programmierte Instruktion . Abstract 
journal. Published by GPI, Berlin. 

Note ; As is obvious from the titles, ”AV Communication Review" and 
"Audiovisual Instruction" arc designed to cover the total audio-visual 
field, but these journals have published much material of interest in the 
area of didactic programming. Several differences in style of presentation 
are seen in the group of journals mentioned above. For instance, "AV 
Communication Review", and "Programmed Learning" favor strict 
scientific reports, whereas "NSPI Journal", for example, has the style 
of an information bulletin, in which news and brief notes of various kinds 
take up more space. The journal activity has been very lively, but it has 
also shown signs of instability. Several of the "early" journals have already 
been discontinued (e. g. , AID, Automated Teaching Bulletin, Deutsche 
Lehrprogramme, Journal of Programmed Instruction, and Programmed 
Instruction). - For more detailed information on U.S. periodicals within 
the area of educational technology, see: Assmann, I. Bibliographie 
amerikanischer Zeitschriften aus dem Bereich der Bildungstechnologie . 
Programmiertes Lernen , 1970, 7^, 174-179. 
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3. SPECIAL" TOPIC PUBLICATIONS 

3. 1 The Pre<»Writin;y Process: Introductory 

Analyses and Decisions (’’System Analysis") 



3.1.1 Analyses of terminal objectives, target populations, and 
subject-matter structure 

Bjerstedt, A. Goal seeking, goal focusing, and goal adjustment. 

Progr. Learn, educ. Technol. , 1970, 268-279. 

Bloom, B.J. (Ed.) Taxonomy of educational objectives . New York: 

Longmans, 1956. 

Boeckmann, K. Basaltext und operationale Lernzicldefinition - Eine 
vergleichende Betrachtung ihrer Mbglichkeiten. In: B. Rollett & 

K. Weltner (Eds. ) Fortschnete und Ergebnisse der Unterrichtstechno- 
logie . Miinchen: Ehrenwirth, 1971. Pp. 17-25. 

Bullock, D.H. Strukturanalyo av en undervisningsenhet. In: E. Wallin (Ed.), 
Unde rvisning - konst eller teknik ? Stockholm: Almqvist & Wiksell, 1968. 
P^. 74-87“ 

Davies, I. K. The mathetics style of programming. In; K. Bung (Ed. ), 
Programmed learning and the language laboratory (I). London: Longmac, 
1968. Pp. 29-50. 

Evans, J.I., Homm<^> L. E. ^ Glaser, B,.. The Ruleg system for the 

construction of programmed verbal learning sequences. J. educ . Res . , 
1962, 513-518. 

Flechsig, K. -H. Probleme der Entscheidung iiber Lernziele. Programmiertes 
Le men , 1970, 7 _, 1-32. 

Frank, H. & Graf, K.D. ALZUDI - Beispiel einer formalen Didaktik. 

Z. erziehungswiss. Forsch. , 1967, ^ (1), 27-34. 

Gagne, R. M. The analysis of instructional objectives for the design of 
instruction. In: R. Glaser (Ed. ), Teaching machines and programed 
learning, II. Washington, D. C. : NEA, 1965^ Pp^ -65. 

Glaser, R. Some research problems in automated instruction: instructional 
programming and subject-matter structure. In: J.E. Coulson (Ed.), 
Programmed learmng and computer-based instruction. New York: 

Wiley, 1962. Pp. ^>7-85. 

Hartley, J. Factors affecting the efficiency of learning from programmed 
instruction. (I). Visual Education , 1971, May, 33-35. 

Kibler, R. J. , Barker, L. L. & Miles, D. T. Behavioral objectives and 
instruction . Boston: Allyn & Bacon, 1970. 

Leith, G.O.M. Learning and personality. In: W.R. Dunn & C. Holroyd 
(Eds.), Aspects of educational technology, Vol. 2. Methuen, 1969. 

Pp. 101-110. 

Mager, R.F. Preparing objectives for programmed instruction. San 
Francisco: Fearon, 1961. 

von Mentzer, C. H. Studier i empirisk m^lanalys. Stockholm: School of 
Education, 1968. 

Miller, R. B. Task description and analysis. In: R.M. Gagne (Ed.) 
Psychological principles in system development . New York: Holt, 

1962. Pp. 187-228. 
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Popham, W. J. et al. Instructional objectives , Chicago: Rand McNally, 

1969. 

Riedel, H. Psychostruktur : Psychostruktur und Liehrprogrammicrung ^ 
Quickborn: Schnelle, 1967. 

Roe, A. & Moon, H. Analysis of course content for individual learning. 
Automated Teaching Bulletin, I960, (3), 3~11. 

Weltner, K. Zur Bestimmung der subjektiven Information durch Rate tests. 

In: Praxis und Perspektiven des programmie rten Unterrichts, Band II . 
Quickborn: Schnelle, 1967. Pp. 69-74. 

Weltner, K. Information theory and programmed instruction. Rev, educ . 
Cybernetics appl. Linguistics, 1969, ^ (1), 25-41. 

Weltner, K. Informationstheorie und Er ziehungswissenschaft . Quickborn: 
Schnelle, 1970. 

Weltner, K. Liernziele unter dem Aspekt der Informationstheorie. In: 

B. Rollett & K. Weltner (Eds.), Fortschritte und Ergebnisse der 
Unte r richtste chnologie . Munchen: Ehrenwirth, 1971. Pp. 26-35. 

Zielke, W. Stichwortschaubild und Faktenanalyse als Hilfsmittel rationellercr 
Lehrprograme r stellung. In: Gesellschaft fiir Programmie rte Instruktion. 

8. Internationales Symposion . . . : Kurzfas sungen . Berlin: GPI, 1970. 

P. 20. 

3.1.2 Decisions on Presentation Media 

Ahlstrom, K. -G. & Amcoff, S. Feedback functions in teaching machines. 
Scand. J. Psychol . , 1967, 243-249. 

Atkinson, R. C. Learning to read under computer control. Progr. Learn. 
educ. Tcchnol. , 1968, 5, 25-37. 

Bjerstedt, A. Schwierigkeitcn and beobachtungstechnologische Mbglich- 
keiten in der Lehrerausbildung. Z. e rziehungswiss . Forsch. , 1968, 2, 
59-82. 

Brown, J. W. Student response systems. Audiovisual Instruction, 1963, 
214-218. 

Bung, K. Programmed learning and the language laboratory (2): Collected 
pape rs . London: Longmac, 1967. 

Bushnell, D.D. The role of the computer in future instructional systems. 
Washington: DA VI, 1963. 

Goldstein, L.S. & Gotkin, L. G. Teaching machines vs. programed 
textbooks as presentation niodes. Journal of programed Instruction, 

1962, i, 29-36. 

Hayes, A.S. Language laboratory facilities . Washington: U.S. Government 
Printing Office , 1963. 

Hocking, E. Language laboratory and language learning . Washington: NEA, 
1964. 

Holling, K. The feedback classroom in use. In: D. Unwin & J. Leedham (Eds.), 
Aspects of educational technology . London: Methuen, 1967. Pp. 27 5-306. 

Lewis, B.N. Gc Pask, G. The theory and practice of adaptive teaching 

systems. In: R. Glaser (Ed.), Teaching machines and programed learning, 
II. Washington, D. C. : NEA, 1965. Pp. 213-266. 
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Malmquist, E. & Grundin, H. U. Olika former f5r bjudning av programme- 
rat undcrvisningsmaterial. Forsloaings rapporter frdn Statens FbrsQks- 
skola i LinlcOpir Nr 8, 196^ 

Pask, G. Teaching machines. In; Znd International Congress on Cybernetics , 
Sept. 1958. Namur, Belg. : Assoc. Intern, de Cybernetique. Pp. 962-978. 

Pask, G. The teaching machine as a control mechanism. Transactions of 
the Society of Instrument Technology , June i960. Pp. 72-89. 

Richter, H. Lehrautomaten - Beispiele und Entwicklungstcndenzcn. In: 

B. Rollett & K. "Weltner (Eds.), Fortschritte und Ergebnisse der Unter- 
richtstechnologie . Miinchen: Ehrenwirth, 1971. Pp. 173-192. 

Ryans, D. G. , Bushnell, D. D. & Cogs ./ell, J.F. A computer-based 
laboratory for automation in school systems. Santa Monica, Calif. : 

System Development Corp. , 1 9<f2. 

Stolurow, L.M. & Davis, D. Teaching machines and computer-based 

systems. In: R. Glaser (Ed.), Teaching machines and programed learning , 

II. Washington, D. C. : NEA, 1965. Pp. 162-212. 

3. 2 Problems Related to the V/riting Process (^'System Synthesis’ ^ 

3.2.1 The problem of "flow structure " 

Chapman, B. Phases of matlietical procedure. Programed Instruction , 

1963-64, 3 (3-4), 6-9. 

Coulson, J.E. et al. Effects of branching in a computer controlled 
autoinstructional device . J. appl. Psychol . , 1962, 46 , 389*-392. 

Crowder, N. A. On the differences between linear and intrinsic programming. 
Phi Delta F \ppan, 1963, 250-254. 

Davies, I. K. Mathetics - a functional approach. In: D. Unwin & J. Leedham 
(Eds.). Aspects of educational technology . London: Methuen, 1967. 

Pp. 205-216. 

Evans, J. L. , Glaser, R. & Homme, L. E. An investigation of "teaching 
machine" variables using learning programs in symbolic logic. J. cduc. 

Res. , 1962, 433-452. 

Gilbert, Th.F. Mathetics: An explicit theory for the design of teaching 

programmes . (Rev. educ. Cyberneties & a-ppi. Linguistics. Supplement 1.) 
London: Longmac, 1969* 

Graf, K. -D. Algoritmische Zuordningsdidaktik und Dialog -Didaktik. In: 

U. Lehncrt (Ed. ), Elektronischc Datenve rarbeitung in Schule und Ausbildung . 
Miinchen: Oldenbourg, 1970. Pp. 64-73. 

Holland, J. G. & Porter, D. The influence of repetition of incorrectly 

answered items in a teaching-machine program. Journal of the experimental 
Analysis of Behavior , 1961, 4, 305-307. 

Jiven, L.M. & Oreberg, C. The IMU Project: Preliminary plan for investi- 
gating the effects of a system for individualized mathematics teaching. 
Didakometry , No. 22, 1968. 

Larkin, T. C. & Leith, G. O.M. The effects of linear and branching methods 
of programmed instruction on learning and retention of a topic in elemen- 
tary science. Programmed Learning , 1964, 12-16. 

Leith, G. O.M. Conflict and interference. Programmed Learning , 1971, 8, 
41-50. 
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Mager, R. F. & Clark, C. Explorations in student-controlled instruction. 

In: G. D. Ofiesh & W. C. Mcicrhenry (Eds.), Trends in programmed 
instruction. Washington: NEA, 1964. Pp. 235-238. 

Pennington, D. F. & Slack, C. V/. The mathetical design of effective 
lessons. In: S. Margulics & L. D. Eigen (Eds.), Applied programed 
instruction. New York: V/ilcy, 1962. Pp. 298-310. 

Presscy, S. L. Basic unresolved teaching-machine problems. Theory into 
Practice, 1962, J,, 30-37. 

Pressey, S.L. Autopre sentation vs . autoelucidation. Programed Instruction , 
1963, 2 (4), 6-7. 

Roe, A. A comparison of branching methods for programmed learning. 

J. educ. Res . , 1962, 407-416. 

Schroter, G. Das zweistufige Programm als integricrender Ubergang vom 
linearen zum verzweigtcn System. In: B. Rollett (Ed. ), Praxis und 
Theoric des programmic rten Untcrrichtes . Stuttgart: Idett, 1970. 

Pp. 32-36. 

Shettel, H.H. Individual differences in subject matter knowledge and pro - 
grammed instructional format . Pittsburgh: AIR, 1963. 

Silberman, H.F. et al. Fixed sequence versus branching autoinstructional 
methods. . T. educ. Psychol ., 1961, 52 , 166-172. 

Smallwood, R.D. A decision structure for teaching machines. Cambridge. 
Mass. : MIT Pr. , 1962. 

3. 2. 2 The characteristics of the single instructional units 

Barlow, J.A. Conversational chaining in teaching machine programs. 
Psychol. Rep . , I960, 7^, 187-193. 

Bascscu, B. More on multiple-choice frames. Programed Instruction, 1962, 
2 (1), 3. 

Bascscu, B. & Horn, R. E. A response analysis systeri. Programed 
Instruction, 1964, 4 (3), 6-9. 

Brethower, D. M. et al. Programmed learning: A practicum . Ann Arbor, 
Mich. : Ann Arbor Publ. , 1965. 

Eigen, E. D. & Margulies, S. Response characteristics as a function of 

information level. Journal of programed Instruction, 1963, ^ (l)» 45-54. 

Ferraro, D. P. et al. Some considerations on the main features of 

programmed instruction. In. G. E. Teal (Ed.), Programmed instruction 
in industry and education. Stamford, Conn. : Public Service Research, 
1963. Pp. 43-67. 

Goldbeck, R.A. & Campbell, V.N. The effects of response mode and 
response difficulty on programed learning. J. educ. Psychol. , 1962, 
110-118. 

Grace, H.A. Programming and readability. Calif. J. educ. Res., 1963, 

U, 164-166. 

Klaus, D. J. The art of auto-instructional programming. AV Communication 
Review , 1961, 9 ^, 130-142. 

Krumboltz, J.D. & Kiesler, Ch. A. The partial reinforcement paradigm and 
' programed instruction. J. progr. Instr . , 1965, ^(2), 9-14» 
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Krumboltz, J. D. & Weisman, R. G. The effect of overt versus covert 

responding to programed instruction on immediate and delayed retention. 

J. cduc. Psychol. , 1962 , 53 ^ 89-92. 

Ludwig, E. H. Die Tcchnik zur Herstellung von Lehrprogrammen . Ratingcn: 
Hcnn, 1965. 

Margulies, S. Some general rules of frame construction. In: G. D. Ofiesh 
V/. C. L/icierhenry (Eds.), Trends in programmed instruction, 
Washington: NEA, 1964. Pp. 185-188. 

Roc, A. Automated teaching methods using linear programs. J. appl. 

Psychol . . 1962, 198-201. 

Schaefer, H. H. A vocabulary program using ’’language redundancy”. 

Journal of programed Instruction, 1963, Z (3), 9-16. 

Trittipoe, W. J. , Trittipoc, T. G. & Hahn, C. P. The effectiveness of three 
programed-learning frame styles. Journal of programed Instruction, 1963, 
2 (1), 35-43. 



^ The Post- Writing Process (’’System Modification and Evaluation” ) 




AERA-APA-DAVI Joint Committee on Programed Instruction and Teaching 
Machines. Criteria for assessing programed instructional materials. 
Audiovisual Instruction, 1963, 8, 84-89. 

AERA-APA-DAVI Joint Committee on Programmed Instruction and Teaching 
Machines. Recommendations for reporting the effectiveness of program- 
mcd-instruction materials . NSPI J . , 1966, ^(3), 3-9. 

Bjerstedt, A. Mapping the effect- structure of self-instructional materials. 
Programmed Learning , 1965, Z, 99-109. 

Bjerstedt, A. Mapping the pheno- structure of self-instructional materials. 
Progr. Learn, cduc. Technol. , 1967, 4, 87-102. 

Bjerstedt, A. PhS.no -Struktur und Effekt-Struktur didaktischcr Sequenzen: 
BeitrSgc zur Analyse und Bcurtcilung von Lehrprogrammen. In: H. Frank 
(Ed. ), L chrmaschincn in kybcrnctischcr und padagogischcr Sicht, 4. 
Stuttgart: Klett, 1966. Pp. 17-39. 

Bloom^ B.S., Hastings, J. Th. & Madaus, G. F. Handbook on formative and 
summativc evaluation of student learning. New York: McGraw-Hill, 1971. 

Cook, D.A. Studying the performance of a program. Programed Instruction , 
1962 , 2 (2), 4-8. 

Cox, A. ct al. Suggested criteria for selection of programed materials. 

In: Summary Report of the Eighth Lake Qkoboji Audiovisual Leade r ship 
Conference . Iowa City: State Univer. Iowa, 1962. Pp. 80-88. 

Geis, G. L. Some considerations in the evaluation of programs. AV Com - 
munication Review, 1962, 1 0 , 64-69. 

Glaser, R. Instructional technology and the measurement of learning 
outcomes : Some questions . Amcr . Psychologist , 1963, 18 , 519-521. 

Gordon, J. M. The evaluative response: A rate problem. AID , 1963, 3, 240. 

Holland, J. G. Evaluating teaching machines and programs. Teachers College 
Record , 1961, 63 , 56-65. 

Jacobs, P. I. Item difficulty and programed learning. Journal of programed 
Instruction, 1963, ^(2), 21-38. 



124 



- 20 . 16 - 



Jacobs, P. I. , Maicr, M. H. & Stolurow, L. M. A nuidc to evaluating self- 
instructional programs. New York: Holt, 196^"! 

Lumsdainc, A. A. Some problems in assessing instructional programs. 
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